#llm-serving · AI & ML

LLM serving

Inference engines, model servers and the gateways that route requests across model APIs.

This week (Oct 27 – Nov 2), the top-ranked are Strata (+4,674), antirion (+302) and resin (+205). The 151 ranked repos gained 27,414 stars, 49% fewer than the week before.

Updated Nov 2, 12:33 UTC · ranked every hour from GitHub’s daily star counts

Updated 51 min ago

GitHub topic ↗

Emerging this week

Showing 10 of 12

How Upstar ranks

Listed by acceleration against their usual week; the score is the rankings’ own: stars gained in 7 days ÷ √(stars before, minimum 100).

1
Alice-Marx/dsh-model-router Model Router and GALGame plugin for DeepSeek Harness After quiet weeks JavaScript JavaScript · 623 total Healthy
Gained in 7 d +221 Score 11.0 Total 623
2
ekzhang/resin 🧱 Experimental library for building inference engines After quiet weeks Python Python · 368 total Mixed signals
Gained in 7 d +205 Score 16.1 Total 368
3
respanai/respan-gateway Open-source LLM gateway in Rust. One endpoint for OpenAI, Anthropic, Gemini and every other vendor. Any SDK, any model, any tool. Breakout week (~1 a week) Rust Rust · 487 total
Gained in 7 d +158 Score 8.7 Total 487
4
kekzl/imp Run large AI chat models privately on your own NVIDIA RTX 50 or RTX PRO graphics card. On one RTX 5090, Qwen3.8-27B writes about 70 words per second, 1.4x faster than vLLM. Even a model almost four times larger than the card's memory runs. Works with apps made for ChatGPT or Claude. Starts with one… 77× its usual pace (~1 a week) C++ C++ · 124 total Healthy
Gained in 7 d +77 Score 7.7 Total 124
5
bitcloud-ai/Bittune Local-first engineering agent that deploys, benchmarks and tunes vLLM/SGLang inference services on your own GPUs, with auditable run evidence. After quiet weeks TypeScript TypeScript · 161 total Mixed signals
Gained in 7 d +66 Score 6.6 Total 161
6
dayuwang401/spectral-orthogonal-exploration Official code for "Student Guides Teacher: Weak-to-Strong Inference via Spectral Orthogonal Exploration" (SOE, ACL 2026 Oral). A training-free, test-time scaling method that fixes reasoning collapse in LLM mathematical reasoning via orthogonal probing. After quiet weeks Python Python · 134 total
Gained in 7 d +60 Score 6.0 Total 134
7
prabanta-dev/janas 37 tok/s from Qwen3-Next-80B (48 GB) on a 32 GB laptop, 24 without drafts vs 2.8 for llama.cpp; reading 62 vs 16 tok/s. Faster than llama.cpp on all 13 models compared in Q4_K_M. LLMs bigger than your RAM, pure C, CPU or GPU. 5× its usual pace (~20 a week) C C · 574 total
Gained in 7 d +106 Score 4.9 Total 574
8
mozilla-ai/otari Open-source, OpenAI-compatible LLM gateway you run yourself. One endpoint for 40+ providers, with virtual keys, budgets, and usage tracking. 4× its usual pace (~20 a week) Python Python · 809 total Healthy
Gained in 7 d +72 Score 2.7 Total 809
9
christopherowen/spark-ds41f Reproducible Docker/vLLM deployment, tuning, and benchmarks for DeepSeek V4.1 Flash on a switchless three-node or four-node DGX Spark fabric. 5× its usual pace (~6 a week) Python Python · 78 total
Gained in 7 d +31 Score 3.1 Total 78
10
carloslfu/slotstream Run a 105 GB AI model on a Mac that can't hold it. Qwen3.8-Flash-Next 4-bit, streamed from SSD on 16–64 GB Apple Silicon Macs. Fully native Swift, MLX and Metal. Offline, no Python. CLI, Swift library, and Ollama/OpenAI/Anthropic APIs. Works with Claude Code and Codex. 3× its usual pace (~37 a week) Swift Swift · 642 total Healthy
Gained in 7 d +104 Score 4.5 Total 642

Back to the ranking Past weeks: Oct 26 Oct 19