Upstar classe toutes les heures les projets GitHub qui montent le plus vite, dans 69 thèmes. Voir les thèmes

#llm-serving · IA & ML

Inférence & passerelles LLM

Moteurs d’inférence, serveurs de modèles et passerelles qui répartissent les requêtes entre les API de modèles.

Cette semaine (27 oct. – 2 nov.), en tête : Strata (+4 667), antirion (+300) et resin (+199). Les 150 projets classés ont gagné 27 323 étoiles, 49 % de moins que la semaine précédente.

Mis à jour le 2 nov., 13:34 UTC · classé toutes les heures à partir des étoiles quotidiennes de GitHub

Mis à jour il y a 23 min

Sujet GitHub ↗ (nouvel onglet)

Classement de la semaine

50 sur 151 affichés

Comment Upstar classe

Classement complet, avec les dépôts masqués par défaut. Revenir au classement

  1. 1
    Niko1221/Strata Qwen3.8-Flash-Next on any consumer hardware: one-click install for Windows / Linux. Strata inference engine, OpenAI/Anthropic API on localhost, optional image input. C++ C++ · 38,1 k au total
    Gagnées en 7 j +4,7 k Score 25,5 Total 38,1 k
  2. 2
    eliasbiondo/antirion 🕷️ The fastest open-source AI Gateway and LLM Gateway — one binary with built-in console to route, secure, budget and observe every LLM call across OpenAI, Anthropic, Gemini, Bedrock and more. Rust Rust · 622 au total Repéré à 95 étoiles
    Gagnées en 7 j +300 Score 16,7 Total 622
  3. 3
    ekzhang/resin 🧱 Experimental library for building inference engines Python Python · 368 au total En bonne santé
    Gagnées en 7 j +199 Score 15,3 Total 368
  4. 4
    StartLuxLabs/StartLux-Prism Fast, exact inference for StartLux-Decision models on NVIDIA GPUs: the released GGUF files on int8 and 4-bit tensor cores, behind the /v1/systemone API, with up to 5.3× lower latency than vLLM in FP8 and bit-identical answers. Python Python · 238 au total
    Gagnées en 7 j +148 Score 14,8 Total 238
  5. 5
    jesdga95/basalt Local inference engine for Qwen3.8-Flash-Next on NVIDIA Blackwell C++ C++ · 448 au total
    Gagnées en 7 j +219 Score 14,5 Total 448
  6. 6
    deepseek-ai/DeepInfer DeepInfer: High-performance, large-scale inference framework for DeepSeek models Python Python · 25 k au total
    Gagnées en 7 j +2,1 k Score 13,8 Total 25 k
  7. 7
    NerverDevs/nervecenter new-api / sub2api 中转站账号管理中心：余额、API Key、用量、倍率、模型比价 | https://nvcenter.cc TypeScript TypeScript · 198 au total
    Gagnées en 7 j +136 Score 13,6 Total 198
  8. 8
    trymirai/uzu A high-performance inference engine for AI models Rust Rust · 3,2 k au total En bonne santé
    Gagnées en 7 j +601 Score 11,8 Total 3,2 k
  9. 9
    Alice-Marx/dsh-model-router Model Router and GALGame plugin for DeepSeek Harness JavaScript JavaScript · 623 au total En bonne santé
    Gagnées en 7 j +220 Score 11,0 Total 623
  10. 10
    skyzh/tiny-llm learn LLM inference system on Apple Silicon for systems engineers: build a tiny vLLM + Qwen Python Python · 5,5 k au total En bonne santé
    Gagnées en 7 j +729 Score 10,5 Total 5,5 k
  11. 11
    VecSzn/PolyStrata Run large AI models on your own PC: Qwen3.8-Flash-Next on Strata's engine, and GLM 5.3 Flash, DeepSeek V4 Flash, LFM2.5, Qwen3.6, Ornith 1.5, Gemma 4, Tiel Coder and MiniCPM-V on a second engine written for PolyStrata. One-click install for Windows and Linux, OpenAI/Anthropic API on localhost. C++ C++ · 760 au total Repéré à 196 étoiles
    Gagnées en 7 j +231 Score 10,0 Total 760
  12. 12
    mw00/project-maya Project Maya - run GLM-5.3-Flash (321B MoE), DeepSeek-V4.1-Flash-1017 (748B MoE) and MiniMax-M3.1-Flash (428B MoE) privately on your own GPU(s), from one card to 16, NVIDIA or AMD, Linux or Windows. Engine, server, dashboard, OpenAI- and Anthropic-compatible API, quant tools. Cuda Cuda · 2,4 k au total Repéré à 762 étoiles
    Gagnées en 7 j +434 Score 9,8 Total 2,4 k
  13. 13
    drawthingsai/local-code-community Source code of Local Code, a private AI agent for Mac, iPad and iPhone. Runs DeepSeek 4.1 on device, with Python, Node.js and a shell built into the app. C C · 90 au total
    Gagnées en 7 j +90 Score 9,0 Total 90
  14. 14
    respanai/respan-gateway Open-source LLM gateway in Rust. One endpoint for OpenAI, Anthropic, Gemini and every other vendor. Any SDK, any model, any tool. Rust Rust · 487 au total
    Gagnées en 7 j +156 Score 8,6 Total 487
  15. 15
    cheaper-inference/OmniRoute Never stop coding. Free MIT AI gateway: one endpoint, 375 providers (150+ free), 1,400+ models Kimi, Claude, GPT, Gemini, GLM, DeepSeek, MiniMax. Works with Claude Code, Codex, Cursor, OpenCode, Cline & Copilot. Quota-aware auto-fallback, RTK+Caveman compression saves 15-95% tokens, MCP/A2A… TypeScript TypeScript · 83,7 k au total En bonne santé
    Gagnées en 7 j +2,4 k Score 8,4 Total 83,7 k
  16. 16
    FlashML-org/FreeToken FreeToken brings datacenter-scale model serving to your desktop. Run massive models locally, fast and efficiently. Python Python · 15,7 k au total En bonne santé
    Gagnées en 7 j +945 Score 7,8 Total 15,7 k
  17. 17
    kekzl/imp Run large AI chat models privately on your own NVIDIA RTX 50 or RTX PRO graphics card. On one RTX 5090, Qwen3.8-27B writes about 70 words per second, 1.4x faster than vLLM. Even a model almost four times larger than the card's memory runs. Works with apps made for ChatGPT or Claude. Starts with one… C++ C++ · 124 au total En bonne santé
    Gagnées en 7 j +77 Score 7,7 Total 124
  18. 18
    routerhubSI/uncensored-ai-api Uncensored AI API and models — one key for Claude, GPT, Gemini, Grok, DeepSeek, GLM and Qwen, with uncensored and abliterated variants where available. OpenAI- and Anthropic-compatible. Pay per token. 509 au total
    Gagnées en 7 j +144 Score 7,5 Total 509
  19. 19
    Calcium-Ion/AstrLink A local AI gateway for AI agents, unifying your subscriptions and API providers with smart routing and on-device privacy protection. Go Go · 524 au total
    Gagnées en 7 j +139 Score 7,1 Total 524
  20. 20
    synchopate/nemotron-30b-cpu-256k Running a 30B MoE LLM on CPU at 30-50+ tok/s with a 256K Context Window via Hybrid Mamba-2 SSM, L3 NTA Slicing, AVX-512 VNNI Tile Repacking & 2MB HugePages Python Python · 70 au total Nouveau
    Gagnées en 7 j +70 Score 7,0 Total 70
  21. 21
    giienew/9router-pro 9Router Pro by Gienetic — Pro edition of 9Router with Prompt Injection (system prompt or chat script per global / API key / model), per-key token quotas, accent themes and an optimized dashboard. JavaScript JavaScript · 83 au total
    Gagnées en 7 j +69 Score 6,9 Total 83
  22. 22
    ThinkWatchProject/ThinkWatch-Lite Local gateway for Claude Code, Codex and other AI clients on macOS, Windows and Linux: switch upstreams without touching clients, keep API keys from relays, cut off malicious tool calls, trace every request. TypeScript TypeScript · 532 au total
    Gagnées en 7 j +133 Score 6,7 Total 532
  23. 23
    bitcloud-ai/Bittune Local-first engineering agent that deploys, benchmarks and tunes vLLM/SGLang inference services on your own GPUs, with auditable run evidence. TypeScript TypeScript · 161 au total Signaux mitigés
    Gagnées en 7 j +65 Score 6,5 Total 161
  24. 24
    rednote-machine-learning/RedKnot Efficient Long-Context LLM Serving with Head-Aware KV Reuse and SegPagedAttention Python Python · 3,1 k au total Signaux mitigés
    Gagnées en 7 j +328 Score 6,3 Total 3,1 k
  25. 25
    dayuwang401/spectral-orthogonal-exploration Official code for "Student Guides Teacher: Weak-to-Strong Inference via Spectral Orthogonal Exploration" (SOE, ACL 2026 Oral). A training-free, test-time scaling method that fixes reasoning collapse in LLM mathematical reasoning via orthogonal probing. Python Python · 134 au total
    Gagnées en 7 j +60 Score 6,0 Total 134
  26. 26
    mnfst/awesome-free-llm-apis List of Permanent Free LLM API (API Keys) JavaScript JavaScript · 12 k au total Signaux mitigés
    Gagnées en 7 j +633 Score 5,9 Total 12 k
  27. 27
    Wallawalla47/Infernix C++/CUDA single-GPU inference engine for Qwen3.8 (incl. Qwen3.8-Flash-Next with offloaded experts) on the RTX 5090; grew from NInfer C++ C++ · 218 au total
    Gagnées en 7 j +70 Score 5,8 Total 218
  28. 28
    cablate/model-relay Per-window model switcher, subagent router, and automatic 429 failover for Claude Code — switch any window with /provider, delegate subagents across models, and keep Remote Control & statuslines working. JavaScript JavaScript · 103 au total
    Gagnées en 7 j +57 Score 5,7 Total 103
  29. 29
    01men/ybkkasi One control plane for every AI asset — MCP servers, skills, agents, AI apps. Self-hosted ledger with lifecycle state machine, 85 permission points, department × agent × skill cost attribution, and audit trail. MIT, zero runtime deps (Node 22 + node:sqlite). 企业 AI 资产治理。 JavaScript JavaScript · 58 au total
    Gagnées en 7 j +53 Score 5,3 Total 58
  30. 30
    musicurgy/delta-inference-engine Delta Inference Engine: DeepSeek-V4.1-Flash on one NVIDIA GPU, natively on Windows (OpenAI-compatible server) C++ C++ · 53 au total Nouveau
    Gagnées en 7 j +53 Score 5,3 Total 53
  31. 31
    shuishuipingan/qoder2api-hub Qoder国际国内多账号反代网关，Qoder 国际国内双区多账号 OpenAI 兼容网关中枢 — COSY 签名推理、稳定设备指纹防风控、OAuth 设备授权免客户端登录、每日签到与 Pro 福利、后台定时调度器、Web 监控看板，支持 Codex / Claude Code 与标准 OpenAI 客户端 Python Python · 498 au total
    Gagnées en 7 j +104 Score 5,2 Total 498
  32. 32
    nace-ai/kapitsa Blazing fast Apple Silicon optimized inference engine built for Qwen & Nemotron. OpenAI + Anthropic API compatible. Built by folks from nace.ai C++ C++ · 595 au total
    Gagnées en 7 j +113 Score 5,1 Total 595
  33. 33
    syv-ai/HyperQwen Serve large Qwen models fast on the GPUs you actually own. Qwen3.8-27B on a single 24 GB card with vLLM: 127 tok/s single-user (381 when the answer quotes the prompt), ~1,035 tok/s at 64 concurrent, 150k-262k context. vLLM patches, requant pipeline, benchmarks. Python Python · 2,5 k au total En bonne santé
    Gagnées en 7 j +243 Score 5,1 Total 2,5 k
  34. 34
    Generated-Computers/Locals Run many AI agents at once on your Mac, up to 3x faster. Python Python · 447 au total
    Gagnées en 7 j +94 Score 5,0 Total 447
  35. 35
    martina9516/uncensored-ai-models Uncensored AI models from RouterHub — the website for one-key API access to Claude, GPT, Gemini, Grok, DeepSeek, GLM and Qwen, with uncensored and abliterated variants where available. OpenAI- and Anthropic-compatible. Pay per token. 108 au total Nouveau
    Gagnées en 7 j +50 Score 5,0 Total 108
  36. 36
    prabanta-dev/janas 37 tok/s from Qwen3-Next-80B (48 GB) on a 32 GB laptop, 24 without drafts vs 2.8 for llama.cpp; reading 62 vs 16 tok/s. Faster than llama.cpp on all 13 models compared in Q4_K_M. LLMs bigger than your RAM, pure C, CPU or GPU. C C · 574 au total
    Gagnées en 7 j +105 Score 4,8 Total 574
  37. 37
    ddalcu/mlx-serve Native LLM inference server for Apple Silicon. OpenAI + Anthropic API compatible. No Python. Zig backend, Swift frontend macOS app with chat, music, voice, video generation. Zig Zig · 2,3 k au total En bonne santé
    Gagnées en 7 j +219 Score 4,8 Total 2,3 k
  38. 38
    ahtx/kindi A custom LLM inference engine for low-cost GPUs: NVIDIA cards from a 1 GB Quadro K420 to an RTX 3080, the AMD Radeon 780M and the CPU. Runs the Bonsai 2B to 27B models and Nemotron with an OpenAI-compatible server. Python Python · 113 au total
    Gagnées en 7 j +47 Score 4,7 Total 113
  39. 39
    T-Crypt/berth Berth is a lightweight GPU resource manager written in Go. It controls which applications get access to your GPU, when they get it, and how that access is handed off. Go Go · 279 au total
    Gagnées en 7 j +66 Score 4,5 Total 279
  40. 40
    Indras-Mirror/Xolotl Xolotl (formerly Strata-DSV4.1): run DeepSeek-V4.1-Flash, all 552B, as a local coding agent on one gaming PC (RTX 4090 + 90 GB RAM) at up to ~27 tok/s, 512K context, October 1017 builds included. Plus GLM-5.3-Flash and Qwen3.8. MoE experts tiered across VRAM, pinned RAM and NVMe; OpenAI + Anthropic… C++ C++ · 441 au total
    Gagnées en 7 j +83 Score 4,4 Total 441
  41. 41
    carloslfu/slotstream Run a 105 GB AI model on a Mac that can't hold it. Qwen3.8-Flash-Next 4-bit, streamed from SSD on 16–64 GB Apple Silicon Macs. Fully native Swift, MLX and Metal. Offline, no Python. CLI, Swift library, and Ollama/OpenAI/Anthropic APIs. Works with Claude Code and Codex. Swift Swift · 642 au total En bonne santé
    Gagnées en 7 j +100 Score 4,3 Total 642
  42. 42
    incoai/splash A local inference engine for Apple silicon, built around the model. C++ C++ · 2 k au total
    Gagnées en 7 j +183 Score 4,3 Total 2 k
  43. 43
    MemTensor/MemT-1.0 Official repository for MemT-1.0 Python Python · 501 au total
    Gagnées en 7 j +85 Score 4,2 Total 501
  44. 44
    lyogavin/airllm AirLLM 2.8T LLM inference with single 4GB GPU Jupyter Notebook Jupyter Notebook · 41,6 k au total En bonne santé
    Gagnées en 7 j +824 Score 4,1 Total 41,6 k
  45. 45
    magnitudedev/magnitude Open source inference engine that tunes its kernels on your hardware to find the fastest way to run each model. 2x faster than Ollama. Works on Apple Silicon, NVIDIA, AMD, Intel, or just a CPU. Rust Rust · 7,7 k au total En bonne santé
    Gagnées en 7 j +334 Score 3,9 Total 7,7 k
  46. 46
    chalk-ai/chalk-router-oss chalk-router: an extremely fast & focused LLM router implementation. Rust Rust · 39 au total Nouveau
    Gagnées en 7 j +39 Score 3,9 Total 39
  47. 47
    eagle-hq/EAGLE-4 Official Inference Code of EAGLE-4 Python Python · 1,1 k au total
    Gagnées en 7 j +112 Score 3,5 Total 1,1 k
  48. 48
    WizisCool/oh-my-cpa Self-hosted control plane & usage observability console for CLIProxyAPI (CPA) — AI providers, client keys, streaming telemetry, and model cost snapshots. TypeScript TypeScript · 296 au total
    Gagnées en 7 j +50 Score 3,2 Total 296
  49. 49
    3145544645/gouji 构屎 GOUJI — 纯 C++/CUDA 的三值 mini LLM。2 bit/权重，零 Python；联想记忆工具路由，零训练参数；143 个缺陷与 29 个变异测试记录在案。 C++ C++ · 99 au total
    Gagnées en 7 j +31 Score 3,1 Total 99
  50. 50
    christopherowen/spark-ds41f Reproducible Docker/vLLM deployment, tuning, and benchmarks for DeepSeek V4.1 Flash on a switchless three-node or four-node DGX Spark fabric. Python Python · 78 au total
    Gagnées en 7 j +30 Score 3,0 Total 78
Afficher la suite

Émergents en Inférence & passerelles LLM Semaines passées : 26 oct.19 oct.