👨🏻💻 LLM Engineer Toolkit
This repository contains a curated list of 120+ LLM libraries category wise.
Related Repositories
🚀 LLM Interview Questions and Answers Book
Crack modern LLM and Generative AI interviews with this comprehensive, interview-focused guide designed specifically for ML Engineers, AI Engineers, Data Scientists and Software Engineers.
This book features 100+ carefully curated LLM interview questions, each paired with clear answers and in-depth explanations so you truly understand the concepts interviewers care about. Get the book here.
Use the Coupon Code: LLMQA25 for an exclusive discount (50%) on the book. (Available only for a short period of time).
Stay Updated with Generative AI, LLMs, Agents and RAG.
Join 🚀 AIxFunda free newsletter to get latest updates and interesting tutorials related to Generative AI, LLMs, Agents and RAG.
- ✨ Weekly GenAI updates
- 📄 Weekly LLM, Agents and RAG paper updates
- 📝 1 fresh blog post on an interesting topic every week
Quick links
LLM Training and Fine-Tuning
| Library |
Description |
Link |
| unsloth |
Fine-tune LLMs faster with less memory. |
Link |
| PEFT |
State-of-the-art Parameter-Efficient Fine-Tuning library. |
Link |
| TRL |
Train transformer language models with reinforcement learning. |
Link |
| Transformers |
Transformers provides thousands of pretrained models to perform tasks on different modalities such as text, vision, and audio. |
Link |
| Axolotl |
Tool designed to streamline post-training for various AI models. |
Link |
| LLMBox |
A comprehensive library for implementing LLMs, including a unified training pipeline and comprehensive model evaluation. |
Link |
| LitGPT |
Train and fine-tune LLM lightning fast. |
Link |
| Mergoo |
A library for easily merging multiple LLM experts, and efficiently train the merged LLM. |
Link |
| Llama-Factory |
Easy and efficient LLM fine-tuning. |
Link |
| Ludwig |
Low-code framework for building custom LLMs, neural networks, and other AI models. |
Link |
| Txtinstruct |
A framework for training instruction-tuned models. |
Link |
| Lamini |
An integrated LLM inference and tuning platform. |
Link |
| XTuring |
xTuring provides fast, efficient and simple fine-tuning of open-source LLMs, such as Mistral, LLaMA, GPT-J, and more. |
Link |
| RL4LMs |
A modular RL library to fine-tune language models to human preferences. |
Link |
| DeepSpeed |
DeepSpeed is a deep learning optimization library that makes distributed training and inference easy, efficient, and effective. |
Link |
| torchtune |
A PyTorch-native library specifically designed for fine-tuning LLMs. |
Link |
| PyTorch Lightning |
A library that offers a high-level interface for pretraining and fine-tuning LLMs. |
Link |
| RapidFire AI |
Rapid AI experimentation framework for fine-tuning and post-training LLMs (SFT, DPO, GRPO) with hyperparallelized execution and real-time interactive control. |
Link |
LLM Application Development
Frameworks
| Library |
Description |
Link |
| LangChain |
LangChain is a framework for developing applications powered by large language models (LLMs). |
Link |
| Llama Index |
LlamaIndex is a data framework for your LLM applications. |
Link |
| HayStack |
Haystack is an end-to-end LLM framework that allows you to build applications powered by LLMs, Transformer models, vector search and more. |
Link |
| Prompt flow |
A suite of development tools designed to streamline the end-to-end development cycle of LLM-based AI applications. |
Link |
| Griptape |
A modular Python framework for building AI-powered applications. |
Link |
| Weave |
Weave is a toolkit for developing Generative AI applications. |
Link |
| Llama Stack |
Build Llama Apps. |
Link |
Data Preparation
| Library |
Description |
Link |
| Data Prep Kit |
Data Prep Kit accelerates unstructured data preparation for LLM app developers. Developers can use Data Prep Kit to cleanse, transform, and enrich use case-specific unstructured data to pre-train LLMs, fine-tune LLMs, instruct-tune LLMs, or build RAG applications. |
Link |
Multi API Access
| Library |
Description |
Link |
| LiteLLM |
Library to call 100+ LLM APIs in OpenAI format. |
Link |
| AI Gateway |
A Blazing Fast AI Gateway with integrated Guardrails. Route to 200+ LLMs, 50+ AI Guardrails with 1 fast & friendly API. |
Link |
| Bifrost |
High-performance AI gateway with a unified OpenAI-compatible API, multi-provider routing, automatic fallbacks, load balancing, and observability. |
Link |
Routers
| Library |
Description |
Link |
| OpenRoutiQ |
Explainable, policy-controlled Python router for selecting models, providers, deployments, and reasoning levels, with optional outcome learning and an OpenAI-compatible proxy. |
Link |
| RouteLLM |
Framework for serving and evaluating LLM routers - save LLM costs without compromising quality. Drop-in replacement for OpenAI's client to route simpler queries to cheaper models. |
Link |
Memory
| Library |
Description |
Link |
| mem0 |
The Memory layer for your AI apps. |
Link |
| Memoripy |
An AI memory layer with short- and long-term storage, semantic clustering, and optional memory decay for context-aware applications. |
Link |
| Tree Ring Memory |
Local-first memory lifecycle for AI agents with Rust CLI/TUI, SQLite/FTS recall, audit, consolidation, and forgetting. |
Link |
| Letta (MemGPT) |
An open-source framework for building stateful LLM applications with advanced reasoning capabilities and transparent long-term memory |
Link |
| Memobase |
A user profile-based memory system designed to bring long-term user memory to your Generative AI applications. |
Link |
Interface
| Library |
Description |
Link |
| Streamlit |
A faster way to build and share data apps. Streamlit lets you transform Python scripts into interactive web apps in minutes |
Link |
| Gradio |
Build and share delightful machine learning apps, all in Python. |
Link |
| AI SDK UI |
Build chat and generative user interfaces. |
Link |
| AI-Gradio |
Create AI apps powered by various AI providers. |
Link |
| Simpleaichat |
Python package for easily interfacing with chat apps, with robust features and minimal code complexity. |
Link |
| Chainlit |
Build production-ready Conversational AI applications in minutes. |
Link |
Low Code
| Library |
Description |
Link |
| LangFlow |
LangFlow is a low-code app builder for RAG and multi-agent AI applications. It’s Python-based and agnostic to any model, API, or database. |
Link |
Cache
| Library |
Description |
Link |
| GPTCache |
A Library for Creating Semantic Cache for LLM Queries. Slash Your LLM API Costs by 10x 💰, Boost Speed by 100x. Fully integrated with LangChain and LlamaIndex. |
Link |
LLM RAG
| Library |
Description |
Link |
| FastGraph RAG |
Streamlined and promptable Fast GraphRAG framework designed for interpretable, high-precision, agent-driven retrieval workflows. |
Link |
| Chonkie |
RAG chunking library that is lightweight, lightning-fast, and easy to use. |
Link |
| RAGChecker |
A Fine-grained Framework For Diagnosing RAG. |
Link |
| RAG to Riches |
Build, scale, and deploy state-of-the-art Retrieval-Augmented Generation applications. |
Link |
| BeyondLLM |
Beyond LLM offers an all-in-one toolkit for experimentation, evaluation, and deployment of Retrieval-Augmented Generation (RAG) systems. |
Link |
| SQLite-Vec |
A vector search SQLite extension that runs anywhere! |
Link |
| fastRAG |
fastRAG is a research framework for efficient and optimized retrieval-augmented generative pipelines, incorporating state-of-the-art LLMs and Information Retrieval. |
Link |
| FlashRAG |
A Python Toolkit for Efficient RAG Research. |
Link |
| Llmware |
Unified framework for building enterprise RAG pipelines with small, specialized models. |
Link |
| Rerankers |
A lightweight unified API for various reranking models. |
Link |
| Vectara |
Build Agentic RAG applications. |
Link |
| RapidFire AI |
Rapid AI experimentation framework for RAG and context engineering with hyperparallelized, shard-based execution and real-time interactive control. |
Link |
LLM Inference
| Library |
Description |
Link |
| llama.cpp |
LLM inference in C/C++. |
Link |
| Ollama |
Local LLM inference. |
Link |
| vLLM |
High-throughput and memory-efficient inference and serving engine for LLMs. |
Link |
| TensorRT-LLM |
TensorRT-LLM is a library for optimizing Large Language Model (LLM) inference. |
Link |
| WebLLM |
High-performance In-browser LLM Inference Engine. |
Link |
| LLM Compressor |
Transformers-compatible library for applying various compression algorithms to LLMs for optimized deployment. |
Link |
| LightLLM |
Python-based LLM inference and serving framework, notable for its lightweight design, easy scalability, and high-speed performance. |
Link |
| torchchat |
Run PyTorch LLMs locally on servers, desktop, and mobile. |
Link |
| llmash |
Ollama-compatible server on a llama.cpp fork with CUDA kernels for speculative decoding; keeps Ollama's API and model store. |
Link |
LLM Serving
| Library |
Description |
Link |
| Langcorn |
Serving LangChain LLM apps and agents automagically with FastAPI. |
Link |
| LitServe |
Lightning-fast serving engine for any AI model of any size. It augments FastAPI with features like batching, streaming, and GPU autoscaling. |
Link |
LLM Data Extraction
| Library |
Description |
Link |
| Crawl4AI |
Open-source LLM Friendly Web Crawler & Scraper. |
Link |
| ScrapeGraphAI |
A web scraping Python library that uses LLM and direct graph logic to create scraping pipelines for websites and local documents (XML, HTML, JSON, Markdown, etc.). |
Link |
| Docling |
Docling parses documents and exports them to the desired format with ease and speed. |
Link |
| Llama Parse |
GenAI-native document parser that can parse complex document data for any downstream LLM use case (RAG, agents). |
Link |
| PyMuPDF4LLM |
PyMuPDF4LLM library makes it easier to extract PDF content in the format you need for LLM & RAG environments. |
Link |
| Crawlee |
A web scraping and browser automation library. |
Link |
| MegaParse |
Parser for every type of document. |
Link |
| ExtractThinker |
Document Intelligence library for LLMs. |
Link |
LLM Data Generation
| Library |
Description |
Link |
| DataDreamer |
DataDreamer is a powerful open-source Python library for prompting, synthetic data generation, and training workflows. |
Link |
| fabricator |
A flexible open-source framework to generate datasets with large language models. |
Link |
| Promptwright |
Synthetic Dataset Generation Library. |
Link |
| EasyInstruct |
An Easy-to-use Instruction Processing Framework for Large Language Models. |
Link |
LLM Agents
| Library |
Description |
Link |
| CrewAI |
Framework for orchestrating role-playing, autonomous AI agents. |
Link |
| LangGraph |
Build resilient language agents as graphs. |
Link |
| Agno |
Build AI Agents with memory, knowledge, tools, and reasoning. Chat with them using a beautiful Agent UI. |
Link |
| Agents SDK |
Build agentic apps using LLMs with context, tools, hand off to other specialized agents. |
Link |
| AutoGen |
An open-source framework for building AI agent systems. |
Link |
| Smolagents |
Library to build powerful agents in a few lines of code. |
Link |
| Pydantic AI |
Python agent framework to build production grade applications with Generative AI. |
Link |
| CAMEL |
Open-source multi-agent framework with various toolkits and use-cases available. |
Link |
| BeeAI |
Build production-ready multi-agent systems in Python. |
Link |
| gradio-tools |
A Python library for converting Gradio apps into tools that can be leveraged by an LLM-based agent to complete its task. |
Link |
| Composio |
Production Ready Toolset for AI Agents. |
Link |
| Atomic Agents |
Building AI agents, atomically. |
Link |
| Memary |
Open Source Memory Layer For Autonomous Agents. |
Link |
| Browser Use |
Make websites accessible for AI agents. |
Link |
| OpenWebAgent |
An Open Toolkit to Enable Web Agents on Large Language Models. |
Link |
| Lagent |
A lightweight framework for building LLM-based agents. |
Link |
| LazyLLM |
A Low-code Development Tool For Building Multi-agent LLMs Applications. |
Link |
| Swarms |
The Enterprise-Grade Production-Ready Multi-Agent Orchestration Framework. |
Link |
| ChatArena |
ChatArena is a library that provides multi-agent language game environments and facilitates research about autonomous LLM agents and their social interactions. |
Link |
| Swarm |
Educational framework exploring ergonomic, lightweight multi-agent orchestration. |
Link |
| AgentStack |
The fastest way to build robust AI agents. |
Link |
| Archgw |
Intelligent gateway for Agents. |
Link |
| Flow |
A lightweight task engine for building AI agents. |
Link |
| AgentOps |
Python SDK for AI agent monitoring. |
Link |
| Langroid |
Multi-Agent framework. |
Link |
| Agentarium |
Framework for creating and managing simulations populated with AI-powered agents. |
Link |
| Upsonic |
Reliable AI agent framework that supports MCP. |
Link |
LLM Evaluation
| Library |
Description |
Link |
| Ragas |
Ragas is your ultimate toolkit for evaluating and optimizing Large Language Model (LLM) applications. |
Link |
| Giskard |
Open-Source Evaluation & Testing for ML & LLM systems. |
Link |
| DeepEval |
LLM Evaluation Framework |
Link |
| Lighteval |
All-in-one toolkit for evaluating LLMs. |
Link |
| Trulens |
Evaluation and Tracking for LLM Experiments |
Link |
| PromptBench |
A unified evaluation framework for large language models. |
Link |
| LangTest |
Deliver Safe & Effective Language Models. 60+ Test Types for Comparing LLM & NLP Models on Accuracy, Bias, Fairness, Robustness & More. |
Link |
| EvalPlus |
A rigorous evaluation framework for LLM4Code. |
Link |
| FastChat |
An open platform for training, serving, and evaluating large language model-based chatbots. |
Link |
| judges |
A small library of LLM judges. |
Link |
| Evals |
Evals is a framework for evaluating LLMs and LLM systems, and an open-source registry of benchmarks. |
Link |
| AgentEvals |
Evaluators and utilities for evaluating the performance of your agents. |
Link |
| LLMBox |
A comprehensive library for implementing LLMs, including a unified training pipeline and comprehensive model evaluation. |
Link |
| Opik |
An open-source end-to-end LLM Development Platform which also includes LLM evaluation. |
Link |
| PydanticAI Evals |
A powerful evaluation framework designed to help you systematically evaluate the performance of LLM applications. |
Link |
| UQLM |
A Python package for generation-time, zero-resource LLM hallucination using state-of-the-art uncertainty quantification techniques. |
Link |
| ai-evaluation |
Open-source LLM evaluation framework with 50+ metrics, LLM-as-Judge, guardrail scanners (jailbreak, PII, injection), and AutoEval pipelines with CI/CD support. |
Link |
| StructEval |
Benchmark and evaluation framework for LLM-generated structured outputs across 18 formats, with structural and visual evaluation. |
Link |
LLM Monitoring
| Library |
Description |
Link |
| MLflow |
An open-source end-to-end MLOps/LLMOps Platform for tracking, evaluating, and monitoring LLM applications. |
Link |
| Opik |
An open-source end-to-end LLM Development Platform which also includes LLM monitoring. |
Link |
| LangSmith |
Provides tools for logging, monitoring, and improving your LLM applications. |
Link |
| Weights & Biases (W&B) |
W&B provides features for tracking LLM performance. |
Link |
| Helicone |
Open source LLM-Observability Platform for Developers. One-line integration for monitoring, metrics, evals, agent tracing, prompt management, playground, etc. |
Link |
| agenttrace |
Local-first TUI and CLI for monitoring AI coding-agent traces, costs, tokens, latency, and health regressions. |
Link |
| OrcaReplay |
Records a coding-agent run below the harness so model traffic, shell exit codes, file changes and MCP calls share one timeline, then replays it offline or forks it onto another model. |
Link |
| Evidently |
An open-source ML and LLM observability framework. |
Link |
| Phoenix |
An open-source AI observability platform designed for experimentation, evaluation, and troubleshooting. |
Link |
| Observers |
A Lightweight Library for AI Observability. |
Link |
| traceAI |
Open-source OpenTelemetry-native tracing framework for LLM and AI agent applications. Auto-instruments 20+ frameworks (OpenAI, Anthropic, LangChain, LlamaIndex, CrewAI, Bedrock) capturing prompts, tokens, latency, and errors. |
Link |
LLM Prompts
| Library |
Description |
Link |
| PCToolkit |
A Unified Plug-and-Play Prompt Compression Toolkit of Large Language Models. |
Link |
| Selective Context |
Selective Context compresses your prompt and context to allow LLMs (such as ChatGPT) to process 2x more content. |
Link |
| LLMLingua |
Library for compressing prompts to accelerate LLM inference. |
Link |
| betterprompt |
Test suite for LLM prompts before pushing them to production. |
Link |
| Promptify |
Solve NLP Problems with LLMs & easily generate different NLP Task prompts for popular generative models like GPT, PaLM, and more with Promptify. |
Link |
| PromptSource |
PromptSource is a toolkit for creating, sharing, and using natural language prompts. |
Link |
| DSPy |
DSPy is the open-source framework for programming—rather than prompting—language models. |
Link |
| Py-priompt |
Prompt design library. |
Link |
| Promptimizer |
Prompt optimization library. |
Link |
LLM Structured Outputs
| Library |
Description |
Link |
| Instructor |
Python library for working with structured outputs from large language models (LLMs). Built on top of Pydantic, it provides a simple, transparent, and user-friendly API. |
Link |
| XGrammar |
An open-source library for efficient, flexible, and portable structured generation. |
Link |
| Outlines |
Robust (structured) text generation |
Link |
| Guidance |
Guidance is an efficient programming paradigm for steering language models. |
Link |
| LMQL |
A language for constraint-guided and efficient LLM programming. |
Link |
| Jsonformer |
A Bulletproof Way to Generate Structured JSON from Language Models. |
Link |
| jebadiah-decide |
Typed choice, yes/no and score decisions with a calibrated probability per option from a Jebadiah model, on llama-server, Ollama, LM Studio, vLLM or MLX. |
Link |
LLM Safety and Security
| Library |
Description |
Link |
| JailbreakEval |
A collection of automated evaluators for assessing jailbreak attempts. |
Link |
| EasyJailbreak |
An easy-to-use Python framework to generate adversarial jailbreak prompts. |
Link |
| Guardrails |
Adding guardrails to large language models. |
Link |
| LLM Guard |
The Security Toolkit for LLM Interactions. |
Link |
| Gorget |
Maintained continuation of LLM Guard: prompt injection, PII and toxicity scanners that run without PyTorch. |
Link |
| AuditNLG |
AuditNLG is an open-source library that can help reduce the risks associated with using generative AI systems for language. |
Link |
| NeMo Guardrails |
NeMo Guardrails is an open-source toolkit for easily adding programmable guardrails to LLM-based conversational systems. |
Link |
| Garak |
LLM vulnerability scanner |
Link |
| DeepTeam |
The LLM Red Teaming Framework |
Link |
| LangFair |
Use-case-specific LLM bias/fairness assessments |
Link |
LLM Embedding Models
| Library |
Description |
Link |
| Sentence-Transformers |
State-of-the-Art Text Embeddings |
Link |
| Model2Vec |
Fast State-of-the-Art Static Embeddings |
Link |
| Text Embedding Inference |
A blazing fast inference solution for text embeddings models. TEI enables high-performance extraction for the most popular models, including FlagEmbedding, Ember, GTE and E5. |
Link |
Others
| Library |
Description |
Link |
| Text Machina |
A modular and extensible Python framework, designed to aid in the creation of high-quality, unbiased datasets to build robust models for MGT-related tasks such as detection, attribution, and boundary detection. |
Link |
| LLM Reasoners |
A library for advanced large language model reasoning. |
Link |
| EasyEdit |
An Easy-to-use Knowledge Editing Framework for Large Language Models. |
Link |
| CodeTF |
CodeTF: One-stop Transformer Library for State-of-the-art Code LLM. |
Link |
| spacy-llm |
This package integrates Large Language Models (LLMs) into spaCy, featuring a modular system for fast prototyping and prompting, and turning unstructured responses into robust outputs for various NLP tasks. |
Link |
| pandas-ai |
Chat with your database (SQL, CSV, pandas, polars, MongoDB, NoSQL, etc.). |
Link |
| LLM Transparency Tool |
An open-source interactive toolkit for analyzing internal workings of Transformer-based language models. |
Link |
| Vanna |
Chat with your SQL database. Accurate Text-to-SQL Generation via LLMs using RAG. |
Link |
| mergekit |
Tools for merging pretrained large language models. |
Link |
| MarkLLM |
An Open-Source Toolkit for LLM Watermarking. |
Link |
| LLMSanitize |
An open-source library for contamination detection in NLP datasets and Large Language Models (LLMs). |
Link |
| Annotateai |
Automatically annotate papers using LLMs. |
Link |
| LLM Reasoner |
Make any LLM think like OpenAI o1 and DeepSeek R1. |
Link |
⭐️ Star History
Please consider giving a star, if you find this repository useful.
