groundingpi/GroundingPI

English | 简体中文

Demo Video · Quick Start · Documentation · Citation

• 2026-10-04: Released the project webpage.
• 2026-10-03: Released the source code, deployment and batch-annotation guides, and full-suite evaluation workflows.
• 2026-10-01: We released the GroundingPI model weights on Hugging Face.
• 2026-09-30: The GroundingPI paper is available on arXiv.

Highlights · Demo · Models · Installation · Deployment Options · Quick Start · vLLM Deployment · Batch Annotation · Tasks and Output Format · Method and Inference Infrastructure · Evaluation · Training · Physical Intelligence · Results · Documentation · License · Citation · Acknowledgement

• A strong grounding foundation model. We introduce GroundingPI, a 4B model built on visual primitives, with a staged training recipe and state-of-the-art grounding performance.
• Transfer toward physical intelligence. Autonomous driving and robotic manipulation evaluations demonstrate the value of this perceptual foundation, including strong ID and OOD performance and improved action-data efficiency.
• Insights into perceptual pretraining and future embodied paradigms. We analyze how pretraining scale and data composition shape grounding and transfer, and discuss implications for System-1 foundation-model design and its complementary role in future embodied systems.

▶ Watch the demo

The grounding checkpoint contains the vision-language model, tokenizer, processor, and custom model code. Downstream action-policy integrations are described in Physical Intelligence.

The serving workflows below target Linux x86_64 and Python 3.12. Start from a compatible accelerator runtime with Torch and vLLM 0.18.x already installed. The serving setup inherits that runtime and installs the bundled Transformers 5.7.0 fork.

git clone https://github.com/groundingpi/GroundingPI.git
cd GroundingPI
python3 -m pip install -r requirements.txt huggingface_hub

Already have a source checkout? Start with cd GroundingPI. Run subsequent commands from the repository root.

requirements.txt installs the lightweight HTTP client, visualization tools, and setup dependencies. Serving, training, and evaluation each use their own environment; pip install -r requirements.txt alone does not install the model runtime. The client does not load weights and requires no Torch installation.

Tested accelerators: NVIDIA B300, B200, H200, H800, and PPU. Use the matching runtime for each accelerator. See Environment Setup for installation details.

The vLLM launcher is the default service. Use the repository adapter and bundled Transformers fork with a compatible accelerator runtime.

All routes expose an OpenAI-compatible image + text API and produce structured visual grounding. Prompts cover referring expressions, object localization, text-region grounding, document layout, and point localization. See Tasks and Output Format for the prompt and coordinate contract. For processing an image collection, start with Batch Annotation.

hf download GroundingPI/GroundingPI --local-dir weights/vlm

# In a compatible NVIDIA GPU runtime with vLLM 0.18.x available:
python3 run.py setup serve
python3 run.py serve

For PPU, replace the setup command with python3 run.py setup serve --platform ppu inside the matching vendor runtime image, then run python3 run.py serve. The installer records the platform, and the launcher selects the corresponding configuration.

Download the complete model package. The vLLM adapter uses a separate overlay and reuses the original weight shards; use a new overlay output directory when switching checkpoints, as described in the Inference Guide.

Keep the service running. In a second terminal, use the environment where you installed requirements.txt and replace your_image.jpg with your image:

from grounding_pi import GroundingPi, visualize

client = GroundingPi(
    base_url="http://127.0.0.1:8000/v1",
    model="groundingpi",
)

# Referring-expression grounding
result = client.predict("your_image.jpg", "the red car", task="bbox")
print(result.to_dict())
if result.valid:
    visualize("your_image.jpg", result).save("prediction.png")

# Point localization
point = client.predict(
    "your_image.jpg", "the center of the red car", task="point"
)
print(point.to_dict())

The command-line example saves result.json and, for valid output, prediction.png. Choose a new output directory for each run:

python3 examples/predict.py \
  --image your_image.jpg --phrase "the red car" \
  --task bbox --output outputs/car

See Examples and the client implementation for more usage details.

The default GroundingPI service uses vLLM's Transformers backend with the repository's model and processor adapter. Start from Linux x86_64 / Python 3.12 with accelerator-compatible Torch and vLLM 0.18.x already installed. The setup inherits that runtime and installs the bundled Transformers 5.7.0 fork.

For NVIDIA GPUs, run from the repository root:

hf download GroundingPI/GroundingPI --local-dir weights/vlm
python3 run.py setup serve --platform gpu
python3 run.py serve --config configs/release/vlm_vllm_gpu.yaml

For PPU, use the matching vendor image and PPU build of vLLM, then replace the two commands after download with:

python3 run.py setup serve --platform ppu
python3 run.py serve --config configs/release/vlm_vllm_ppu.yaml

If the serving environment is already prepared, skip setup. Setup requires a new environment directory; use the same --venv on setup and serving when choosing a different directory. Once the server is ready, check its model list in another terminal:

curl --fail http://127.0.0.1:8000/v1/models

The endpoint is http://127.0.0.1:8000/v1, with model ID groundingpi. Defaults are BF16, eager execution, TP=1, one active sequence, 16,384 context tokens, and 0.7 accelerator-memory utilization. Each request accepts one image; video is disabled. The adapter reuses the checkpoint's weights through a separate serving overlay.

For custom /chat/completions requests, set skip_special_tokens: false and spaces_between_special_tokens: false to preserve GAM's adjacent coordinate tokens. Use GAM evaluation mode. See the vLLM Deployment Guide for a complete single-image API example, configuration overrides, and runtime checks.

Use the JSONL batch guide to annotate image collections with per-image grounding prompts. The batch example saves raw responses, parsed coordinates, completion status, and usage; successful items can be skipped when resuming the same inputs and request configuration.

With a service running and your input manifest prepared:

python3 examples/batch_predict.py \
  --input requests.jsonl --output predictions.jsonl \
  --base-url http://127.0.0.1:8000/v1 --model groundingpi

The supplied serving profiles use one active sequence. Batch processing here means sequential image requests with durable output. For multiple accelerators, split the manifest across independent service replicas and use a separate output file per worker. Inspect truncated or invalid results before using predictions as annotations.

The models accept one image and a text instruction. The convenience client's predict() method constructs referring-box and referring-point prompts. For other tasks, send the corresponding prompt to the service's OpenAI-compatible /v1/chat/completions endpoint.

Given reference boxes <|box_start|><100><200><500><650><|box_end|> indicating one or more objects, find all similar objects in the image and output their bounding boxes.

All released checkpoints use the GAM spatial-token protocol, with integer coordinates from 0 to 999. Boxes contain (x1, y1, x2, y2); points contain (x, y). A missing target is represented by None.

<|object_ref_start|>car<|object_ref_end|><|box_start|><100><200><500><650><|box_end|>
<|object_ref_start|>car center<|object_ref_end|><|box_start|><300><425><|box_end|>
<|object_ref_start|>absent object<|object_ref_end|><|box_start|>None<|box_end|>

Use the checkpoint's tokenizer, processor, and chat template. Custom HTTP requests should keep skip_special_tokens=false, preserve adjacent coordinate tokens without added spaces, and send the image as a base64 data URI in an image_url content part alongside the text prompt.

GroundingPI combines a MoonViT-V2 (Kimi K3) visual encoder, a learnable multimodal projector, and a Qwen3-4B language backbone.

The default service uses vLLM, BF16, one image per request, a 16,384-token context limit, and tensor parallel size 1. The custom adapter preserves the model's native spatial-token interface while vLLM manages execution and KV caching.

The supplied launcher uses eager execution; CUDA Graph is disabled in this recipe. A native Transformers reference service is also available. See Inference for custom configurations and reference-backend usage.

The GroundingPI evaluation suite covers the 34 benchmarks reported in the paper. The Evaluation Guide provides the data link, path setup, the exact 34-task selection, input validation, and full-suite execution.

The shared evaluator supports 7 modes:

Use GAM mode for all three released checkpoints. Start the model service and configure the data paths before running:

python3 run.py setup eval
python3 run.py eval --config configs/eval/gam.yaml

The shipped recipes are 8-sample smoke tests for selected tasks. For the complete 34-benchmark suite, follow the full evaluation walkthrough: it selects the paper's task list, uses limit: null, and writes results under a fresh run_id. Evaluation connects to an existing service and does not start or switch its decoder.

Training runs in its own environment. Prepare the complete model files, tokenizer, validated input caches, and manifests before launching. The supplied training environment targets the matching PPU vendor image. See Environment Setup for the matching runtime and dependencies.

The native recipe supports separate learning rates for the language model, vision encoder, and projector. Set their freeze flags to select which components to train. Keep runtime.expected_nodes, expected_gpus_per_node, and expected_world_size consistent with the launch topology. The effective global batch size is the per-device batch size multiplied by world size and gradient accumulation steps.

python3 run.py setup train

# Inspect the configured training command before launching.
.venv-train/bin/python scripts/run.py configs/release/vlm_train.yaml --dry-run

python3 run.py train --config configs/release/vlm_train.yaml

The default recipe uses BF16 and writes checkpoints to outputs/vlm_train/. Input caches must match the model tokenizer and carry the required manifests. See Data Preparation; the repository does not provide a generic JSONL-to-training-cache converter.

Point checkpoint.resume_from_checkpoint in the launch YAML, or training.resume_from_checkpoint in the native YAML, to a complete training checkpoint. Configure it in one place and rerun the same launch command. A checkpoint with optimizer, scheduler, and training state is required to continue an interrupted run.

See Training for the detailed configuration and checkpoint workflow.

The vla/ directory contains companion backbone-comparison workflows built on StarVLA and OpenWAM, with separate policy-training and evaluation configurations:

The Hugging Face release is a grounding vision-language model. The supplied comparison recipes use the backbones listed in the VLA Guide and require their own environments and policy checkpoints; they do not provide a direct action-policy adapter for the released GroundingPI checkpoint.

Full benchmark definitions, comparisons, and downstream experiments are in the paper and supplementary material.

GroundingPI/
├── run.py                  # Workflow launcher
├── grounding_pi/             # Lightweight HTTP client and visualization
├── configs/                # Serving, training, and evaluation recipes
├── infer/                  # Model services
├── train/                  # Training workflows
├── eval/                   # Tasks, prompts, requests, and metrics
├── models/                 # Model definitions
├── examples/               # Prediction examples
├── environments/           # Environment installers
└── docs/                   # Detailed guides

Use python3 run.py --help to inspect the command-line interface. Full model workflows run from the source tree; the Python package installs the independent HTTP client.

Original project contributions are available under the Apache License 2.0, with no additional restrictions imposed by this project. This grant covers only rights held by the contributing authors.

Third-party material retains its applicable licenses, including the Kimi K3 License for Kimi-derived material and applicable derivative works. These upstream conditions remain in force. See Third-party Notices for component attribution and the released model's license scope for the model package.

The physical-intelligence integrations retain the terms in vla/LICENSE and their third-party notices.

If this work supports your research, please cite:

@misc{yu2026groundingpigroundingfoundationmodel,
  title = {{GroundingPI}: A Grounding Foundation Model towards Physical Intelligence with Visual Primitives},
  author = {Qize Yu and Lianrui Fan and Boyu Chen and Jiaqi Liang and Xini Ding and Yue Chen and Zetian Song and Yuran Wang and Yi Zou and Kaixuan Wang and Tianxing Chen and Wenxuan Song and Bohan Zhou and Mingleyang Li and Siqiao Huang and Yuqi Ye and Caigao Jiang and Wei Wei and Ruihai Wu and Hang Zhang and Yixiao Ge and Shuchang Zhou and Shilong Liu and Xianming Liu and Ping Luo and Shiyu Huang},
  year = {2026},
  eprint = {2609.39601},
  archivePrefix = {arXiv},
  primaryClass = {cs.CV},
  url = {https://arxiv.org/abs/2609.39601},
}

We thank the teams behind Rex-Omni and LocateAnything for sharing their work and open-source implementations.