groundingpi/GroundAnything

Demo Video · Quick Start · Documentation · Citation

• 2026-10-04: Released the project webpage.
• 2026-10-03: Released the source code, deployment and batch-annotation guides, and full-suite evaluation workflows.
• 2026-10-01: We released the GroundAnything and GroundAnything-VLM model weights on Hugging Face.
• 2026-09-30: The GroundAnything paper is available on arXiv.

Highlights · Demo · Models · Installation · Deployment Options · Quick Start · vLLM Deployment · Batch Annotation · Tasks and Output Format · Method and Inference Infrastructure · Evaluation · Training · Results · Documentation · License · Citation · Acknowledgement

• Unlocking diffusion for broad visual grounding. We introduce GroundAnything, a 4B foundation model that unifies diverse grounding tasks with precise localization and parallel decoding, trained through grounding pretraining, AR-to-diffusion conversion with joint objectives, supervised fine-tuning, and GRPO-based reinforcement learning.
• State-of-the-art grounding at 4B scale. Across 30 benchmarks, GroundAnything surpasses the prior overall state of the art among similarly sized AR models and outperforms the larger Qwen3.7-Max. GroundAnything-VLM establishes a new overall state of the art at this scale and remains competitive with GPT-6 Astra.
• Systematic acceleration studies. We analyze decoding strategies in detail, compare with MTP-based generation, and evaluate progressive infrastructure optimizations for practical deployment.

▶ Watch the demo

▶ Parallel Decoding

GroundAnything's main results use entropy-guided decoding. GroundAnything-VLM is a separate autoregressive checkpoint. Both use the GAM evaluation mode.

The serving workflows below target Linux x86_64 and Python 3.12. The serving setup installs the bundled custom SGLang engine and its pinned dependencies. The supplied serving profile uses Torch 2.9.1, Transformers 5.5.4, Triton 3.5.1, sgl-kernel 0.3.20, and FlashInfer 0.5.3.

git clone https://github.com/groundingpi/GroundAnything.git
cd GroundAnything
python3 -m pip install -r requirements.txt huggingface_hub

Already have a source checkout? Start with cd GroundAnything. Run subsequent commands from the repository root.

requirements.txt installs the lightweight HTTP client, visualization tools, and setup dependencies. Serving, training, and evaluation each use their own environment; pip install -r requirements.txt alone does not install the model runtime. The client does not load weights and requires no Torch installation.

Tested accelerators: NVIDIA B300, B200, H200, H800, and PPU. Use the matching runtime for each accelerator. See Environment Setup for installation details.

Use the repository's patched engines and launchers. The new vLLM adapter is for GroundAnything-VLM only and has CPU-level checks; end-to-end accelerator validation is pending. DLM decoding stays on SGLang.

All routes expose an OpenAI-compatible image + text API and produce structured visual grounding. Prompts cover referring expressions, object localization, text-region grounding, document layout, and point localization. See Tasks and Output Format for the prompt and coordinate contract. For processing an image collection, start with Batch Annotation.

Download the complete DLM release and start the service:

hf download GroundingPI/GroundAnything --local-dir weights/dlm_bundle
python3 run.py setup serve
python3 run.py serve --decoder denoise

The published checkpoint is already a complete model bundle, so it does not require prepare-model. The supplied SGLang installer is a CUDA/GPU recipe; PPU requires its matching platform runtime.

Stop the existing DLM service, then reuse the same weights with:

python3 run.py serve --decoder speculative

Download the separate autoregressive checkpoint. If the serving environment has not yet been prepared, run python3 run.py setup serve first.

hf download GroundingPI/GroundAnything-VLM --local-dir weights/vlm
python3 run.py serve --config configs/release/vlm_sglang.yaml

Keep the service running. In a second terminal, use the environment where you installed requirements.txt and replace your_image.jpg with your image:

from grounding_anything import GroundingAnything, visualize

client = GroundingAnything(
    base_url="http://127.0.0.1:8101/v1",
    model="groundinganything",
)

# Referring-expression grounding
result = client.predict("your_image.jpg", "the red car", task="bbox")
print(result.to_dict())
if result.valid:
    visualize("your_image.jpg", result).save("prediction.png")

# Point localization
point = client.predict(
    "your_image.jpg", "the center of the red car", task="point"
)
print(point.to_dict())

For GroundAnything-VLM, use the same client with base_url="http://127.0.0.1:8102/v1" and model="groundinganything-vlm".

The command-line example saves result.json and, for valid output, prediction.png. Choose a new output directory for each run:

python3 examples/predict.py \
  --image your_image.jpg --phrase "the red car" \
  --task bbox --output outputs/car

For the VLM service, append --base-url http://127.0.0.1:8102/v1 --model groundinganything-vlm.

See Examples and the client implementation for more usage details.

GroundAnything-VLM also has an opt-in vLLM Transformers-backend adapter for autoregressive grounding. Its dedicated .venv-vllm environment uses Linux x86_64 / Python 3.12, a preinstalled accelerator-compatible Torch / vLLM 0.18.x runtime, and the bundled Transformers 5.7.0 fork. The default SGLang environment remains separate.

For NVIDIA GPUs, run from the repository root:

hf download GroundingPI/GroundAnything-VLM --local-dir weights/vlm
python3 run.py setup vllm --platform gpu
python3 run.py serve --engine vllm --platform gpu

For PPU, run inside the matching vendor runtime image and replace gpu with ppu in both setup and serving commands. Skip setup if .venv-vllm is already prepared. Use the same --venv on both commands to select another fresh environment.

The API is http://127.0.0.1:8102/v1, with model ID groundinganything-vlm. Stop an existing SGLang VLM service on that port before starting vLLM, or choose a different port in the vLLM release configuration. Check readiness in another terminal:

curl --fail http://127.0.0.1:8102/v1/models

Defaults are BF16, eager execution, TP=1, one active sequence, 16,384 context tokens, and one image per request. Custom requests must include skip_special_tokens: false and spaces_between_special_tokens: false. Evaluate with GAM mode using configs/eval/vlm_vllm.yaml.

GroundAnything's DLM entropy-guided and self-speculative decoders use the custom SGLang backend; the vLLM route accepts only the separate VLM checkpoint. This new adapter has CPU-level checks; end-to-end accelerator validation is pending. See the vLLM Deployment Guide for GPU/PPU setup, a complete image request, and configuration details.

Use the JSONL batch guide to annotate image collections with per-image grounding prompts. The batch example saves raw responses, parsed coordinates, completion status, and usage; successful items can be skipped when resuming the same inputs and request configuration.

With a service running and your input manifest prepared:

python3 examples/batch_predict.py \
  --input requests.jsonl --output predictions.jsonl \
  --base-url http://127.0.0.1:8101/v1 --model groundinganything

The supplied serving profiles use one active sequence. Batch processing here means sequential image requests with durable output. For multiple accelerators, split the manifest across independent service replicas and use a separate output file per worker. Inspect truncated or invalid results before using predictions as annotations.

The models accept one image and a text instruction. The convenience client's predict() method constructs referring-box and referring-point prompts. For other tasks, send the corresponding prompt to the service's OpenAI-compatible /v1/chat/completions endpoint.

Given reference boxes <|box_start|><100><200><500><650><|box_end|> indicating one or more objects, find all similar objects in the image and output their bounding boxes.

All released checkpoints use the GAM spatial-token protocol, with integer coordinates from 0 to 999. Boxes contain (x1, y1, x2, y2); points contain (x, y). A missing target is represented by None.

<|object_ref_start|>car<|object_ref_end|><|box_start|><100><200><500><650><|box_end|>
<|object_ref_start|>car center<|object_ref_end|><|box_start|><300><425><|box_end|>
<|object_ref_start|>absent object<|object_ref_end|><|box_start|>None<|box_end|>

Use the checkpoint's tokenizer, processor, and chat template. Custom HTTP requests should keep skip_special_tokens=false, preserve adjacent coordinate tokens without added spaces, and send the image as a base64 data URI in an image_url content part alongside the text prompt.

The variants share a MoonViT-V2 (Kimi K3) visual encoder, a multimodal projector, and a Qwen3-4B language architecture.

The release recipe uses block size 32, sub-block size 4, and entropy threshold 0.8. Each physical block contains a known anchor and 31 masked positions.

After image/query prefill, GroundAnything denoises response blocks with bidirectional attention and caches the completed prefix. Sub-blocks are processed from left to right. Our default Entropy-Guided Decoding commits masked positions with $H_j\leq\tau$, where $H_j=-\sum_v p_j(v)\log p_j(v)$ is the entropy of the unmodified token distribution. If none qualifies, the lowest-entropy position is committed to ensure progress. Committed tokens remain fixed, and a causal pass constructs the completed block's cache without AR verification.

The shared weights also support diffusion drafting with causal verification, accepting the longest matching prefix. Verification accepts only the longest consecutive draft prefix agreeing with the causal predictions. The first disagreement ends acceptance; later coincidental matches are discarded. Rejected suffix states are removed from the cache.

Self-speculation verifies against the converted model's causal branch. Exact greedy verification preserves that branch's outputs, not necessarily the outputs of the separately trained GroundAnything-VLM.

The documented --decoder speculative route uses linear drafting and greedy verification.

The implementation progresses from Native PyTorch (Eager) to SGLang (Eager), CUDA Graph replay, and selective FP8 execution. Denoising and causal passes retain their respective attention and cache semantics. FP8 applies to eligible language-model linear operations, with the remaining components kept in BF16. These changes reduce execution overhead or arithmetic cost; quantization can still alter logits and decoding decisions.

The default DLM service uses BF16, Triton attention, eager execution, one GPU, one active request, and two queued requests. Client concurrency queues requests; use independent replicas for higher service concurrency.

Use the bundled engine and serving profile. Installing upstream SGLang alone does not provide these model and decoding integrations. See Inference for configuration and the native reference route.

The GroundAnything evaluation suite covers the 30 benchmarks reported in the paper. The Evaluation Guide provides the data link, path setup, the exact 30-task selection, input validation, and full-suite execution.

The shared evaluator supports 7 modes:

Use GAM mode for all three released checkpoints. Start the model service and configure the data paths before running:

python3 run.py setup eval
# Select the route matching the service that is already running.
python3 run.py eval --decoder denoise
python3 run.py eval --decoder speculative
python3 run.py eval --config configs/eval/vlm.yaml

The shipped recipes are 8-sample smoke tests for selected tasks. For the complete 30-benchmark suite, follow the full evaluation walkthrough: it selects the paper's task list, uses limit: null, and writes results under a fresh run_id. Evaluation connects to an existing service and does not start or switch its decoder.

Training runs in its own environment. Prepare the complete model files, tokenizer, validated input caches, and manifests before launching. The supplied training recipe targets H800-class CUDA GPUs, with a pinned CUDA toolchain and source-built FlashAttention. See Environment Setup for the matching runtime and dependencies.

Set the base-model path, matching tokenizer, prepared inputs, output directory, batch size, and sequence length in the native configuration. Set the distributed topology and initial checkpoint in the launch configuration. Keep the training mode as DLM; GAM is the evaluation mode.

The supplied stage templates target 8 nodes × 8 devices. Adjust nnodes, nproc_per_node, node_rank, master_addr, and master_port for your cluster, and keep the runtime checks and packing manifest consistent with the chosen world size and batch size.

python3 run.py setup train

# Inspect the configured training command before launching.
.venv-train/bin/python scripts/run.py configs/release/sft1_train.yaml --dry-run

# Run on each node with its corresponding launch configuration.
python3 run.py train --config configs/release/sft1_train.yaml

Training consumes prepared caches and manifests for sampling, length/image checks, response lengths, and packing. These inputs must be prepared before launch; a generic JSONL-to-training-cache converter is not included. See Data Preparation.

Choose one of these settings. To fine-tune the released DLM, use the complete weights/dlm_bundle as the initial DLM checkpoint and set the native model.path to the matching autoregressive base model, with its tokenizer. To resume SFT, update its launch YAML and rerun python3 run.py train --config ...; the top-level --resume shortcut selects the smoke-test recipe.

The Training Guide also covers optional reinforcement learning. After training, export a fresh complete inference bundle using the Inference Guide. The published Hugging Face DLM bundle is already prepared for inference.

Paper and supplementary material.

GroundAnything/
├── run.py                  # Workflow launcher
├── grounding_anything/       # Lightweight HTTP client and visualization
├── configs/                # Serving, training, and evaluation recipes
├── infer/                  # Model services and decoding
├── train/                  # Training workflows
├── eval/                   # Tasks, prompts, requests, and metrics
├── models/                 # Model definitions
├── examples/               # Prediction examples
├── environments/           # Environment installers
└── docs/                   # Detailed guides

Use python3 run.py --help to inspect the command-line interface. Full model workflows run from the source tree; the Python package installs the independent HTTP client.

Original project contributions are available under the Apache License 2.0, with no additional restrictions imposed by this project. This grant covers only rights held by the contributing authors.

Third-party material retains its applicable licenses, including the Kimi K3 License for Kimi-derived material and applicable derivative works. These upstream conditions remain in force. See Third-party Notices for component attribution and the released model's license scope for the model package.

If this work supports your research, please cite:

@misc{yu2026groundanythingreconcilingparalleldecoding,
  title = {{GroundAnything}: Reconciling Parallel Decoding with Precise Visual Grounding at Flash Speed},
  author = {Qize Yu and Lianrui Fan and Bowen Ping and Xini Ding and Zetian Song and Junbo Niu and Kaixuan Wang and Tianxing Chen and Yue Chen and Minghua He and Yuran Wang and Jie Huang and Haojun Zhang and Min Chen and Hao Li and Wenxuan Song and Ruihai Wu and Xianming Liu and Shilong Liu and Shuchang Zhou and Ping Luo and Shiyu Huang},
  year = {2026},
  eprint = {2609.39600},
  archivePrefix = {arXiv},
  primaryClass = {cs.CV},
  url = {https://arxiv.org/abs/2609.39600},
}

We thank the teams behind Rex-Omni and LocateAnything for sharing their work and open-source implementations.