Skill Seekers
English | 简体中文 | 日本語 | 한국어 | Español | Français | Deutsch | Português | Türkçe | العربية | हिन्दी | Русский
🧠 The data layer for AI systems. Skill Seekers turns documentation sites, GitHub repos, PDFs, videos, notebooks, wikis, and more — 18 source types — into structured knowledge assets, ready to power AI Skills (Claude, Gemini, OpenAI), RAG pipelines (LangChain, LlamaIndex, Pinecone), and AI coding assistants (Cursor, Windsurf, Cline). Prepare once, export to 22 targets.
💛 Sponsors
Bronze Sponsors
Sponsor — Bronze
Become a sponsor · GitHub Sponsors
🚀 Quick Start
# 1. Install pip install skill-seekers # 2. Create a skill from any source skill-seekers create https://docs.djangoproject.com/ # Optional: preview how a source will be detected without creating anything skill-seekers detect https://docs.djangoproject.com/ --json # 3. Package it for your AI platform skill-seekers package output/django --target claude
You now have output/django-claude.zip, ready to use.
# Pick a different AI agent for enhancement (default: claude) skill-seekers create https://docs.djangoproject.com/ --agent kimi skill-seekers create https://docs.djangoproject.com/ --agent-cmd "my-custom-agent run"
🛰️ AI-driven project scan
Point scan at a project and an AI agent reads its manifests, README, Dockerfile/CI and sampled source imports — then emits one config per detected framework, plus a <project>-codebase.json for your own code:
skill-seekers scan ./my-react-app --out ./configs/scanned/
# → react.json, vite.json, tailwind.json, jest.json, my-react-app-codebase.json
skill-seekers create ./configs/scanned/react.jsonIf a detection has no existing preset, the AI generates a fresh config; on exit you can optionally publish it back to the community registry.
All 18 source types
skill-seekers create facebook/react # GitHub repository skill-seekers create ./my-project # Local codebase skill-seekers create manual.pdf # PDF skill-seekers create report.docx # Word skill-seekers create book.epub # EPUB skill-seekers create notebook.ipynb # Jupyter skill-seekers create openapi.yaml # OpenAPI/Swagger skill-seekers create presentation.pptx # PowerPoint skill-seekers create guide.adoc # AsciiDoc skill-seekers create page.html # Local HTML (or a whole dir) skill-seekers create feed.rss # RSS/Atom skill-seekers create curl.1 # Man page # Video (YouTube, Vimeo, or local — needs skill-seekers[video]) skill-seekers create --video-url https://www.youtube.com/watch?v=... --name mytutorial skill-seekers create --setup # auto-install GPU-aware visual deps skill-seekers create --space-key TEAM --name wiki # Confluence skill-seekers create --database-id ... --name docs # Notion skill-seekers create --chat-export-path ./slack-export --name team-chat # Slack/Discord
See the Scraping Guide for every source type and its options.
📦 Installation
pip install skill-seekers # Core: scraping, GitHub, PDF, packaging pip install skill-seekers[all-llms] # + every LLM platform pip install skill-seekers[mcp] # + MCP server pip install skill-seekers[all] # Everything
Not sure what you need? Run the wizard: skill-seekers-setup
All installation extras
| Install | Adds |
|---|---|
skill-seekers[gemini] |
Google Gemini support |
skill-seekers[openai] |
OpenAI ChatGPT support |
skill-seekers[all-llms] |
All LLM platforms |
skill-seekers[mcp] |
MCP server for Claude Code, Cursor, etc. |
skill-seekers[video] |
YouTube/Vimeo transcript & metadata extraction |
skill-seekers[video-full] |
+ Whisper transcription & visual frame extraction |
skill-seekers[jupyter] |
Jupyter Notebook support |
skill-seekers[pptx] |
PowerPoint support |
skill-seekers[confluence] |
Confluence wiki support |
skill-seekers[notion] |
Notion pages support |
skill-seekers[rss] |
RSS/Atom feed support |
skill-seekers[chat] |
Slack/Discord chat export support |
skill-seekers[asciidoc] |
AsciiDoc support |
skill-seekers[all] |
Everything |
Video visual deps (GPU-aware): after installing
skill-seekers[video-full], runskill-seekers create --setupto auto-detect your GPU and install the matching PyTorch variant + easyocr.
Prerequisites: Python 3.10+, Git. New here? → Bulletproof Quick Start 🎯
📚 Documentation
| I want to... | Read this |
|---|---|
| Get started quickly | Quick Start — 3 commands to your first skill |
| Understand the concepts | Core Concepts |
| Scrape sources | Scraping Guide — all 18 source types |
| Enhance skills with AI | Enhancement Guide · Enhancement Modes |
| Export skills | Packaging Guide |
| Build workflows | Workflows |
| Look up a command | CLI Reference — all 19 commands |
| Configure | Config Format · Environment Variables |
| Set up MCP | MCP Setup · MCP Reference |
| Integrate with RAG / IDEs | LangChain · RAG Pipelines · Cursor · Windsurf · Cline |
| Handle huge doc sets | Large Documentation — 10K–40K+ pages |
| Understand the architecture | UML Architecture — 14 diagrams |
| Fix a problem | Troubleshooting |
Complete documentation index: docs/README.md
🎯 What you get
| Use case | Output | Powers |
|---|---|---|
| AI Skills | Comprehensive SKILL.md + reference files |
Claude Code, Gemini, GPT |
| RAG pipelines | Chunked documents with rich metadata | LangChain, LlamaIndex, Haystack |
| Vector databases | Pre-formatted data ready for upsert | Pinecone, Chroma, Weaviate, FAISS, Qdrant |
| AI coding assistants | Context files your IDE AI reads automatically | Cursor, Windsurf, Cline, Continue.dev |
Export targets (22)
skill-seekers package output/react --target claude # → Claude Skill (ZIP + YAML) skill-seekers package output/react --target langchain # → LangChain Documents skill-seekers package output/react --target llama-index # → LlamaIndex TextNodes skill-seekers package output/react --target ibm-bob # → IBM Bob skill directory
LLM platforms (12): claude · gemini · openai · minimax · opencode · kimi · deepseek · qwen · openrouter · together · fireworks · markdown
RAG & vector (8): langchain · llama-index · haystack · chroma · faiss · weaviate · qdrant · pinecone
Other (2): atlas · ibm-bob
See the Feature Matrix for per-platform support details.
Why it matters
- ⚡ 99% faster — days of manual data prep → 15–45 minutes
- 🎯 Real skill quality — 500+ line
SKILL.mdfiles with examples, patterns, and guides - 📊 RAG-ready chunks — smart chunking preserves code blocks and context
- 🔄 Multi-source — combine docs + GitHub + PDFs + videos into one knowledge asset
- 🌐 One prep, every target — export to 22 targets without re-scraping
- ✅ Battle-tested — 3,900+ tests, 68 workflow presets, production-ready
✨ Key capabilities
Documentation scraping — SPA discovery, llms.txt, smart categorization
Three-layer discovery for JavaScript SPA sites (sitemap.xml → llms.txt → headless browser rendering), automatic llms.txt detection (10× faster when present), smart topic categorization, and a lenient HTML parser fallback so broken markup still scrapes.
GitHub & codebase analysis (C3.x) — AST parsing, pattern detection, how-to guides
Three-stream architecture: code analysis (AST, design patterns, tests), documentation (README, docs/, wiki), and community (issues, PRs, metadata). The C3.x pipeline adds 10 GoF pattern detectors across 9 languages, usage examples extracted from tests, AI-written how-to guides, config extraction, and architecture overviews.
skill-seekers create ./my-project --preset quick # 1–2 min, surface level skill-seekers create ./my-project --preset standard # balanced (default) skill-seekers create ./my-project --preset comprehensive # deep, exhaustive
→ Pattern Detection · How-To Guides · Test Example Extraction
AI enhancement — API or local agents, 68 workflow presets
Every AI call runs through one transport, in API mode (Anthropic, Google Gemini, OpenAI, Moonshot/Kimi, MiniMax) or LOCAL mode (Claude Code, Kimi Code, Codex, Copilot, OpenCode, custom agents — no API costs). Control depth with --enhance-level 0-3 and pick an agent with --agent.
Unified multi-source scraping — combine many sources into one skill
One config can pull documentation, GitHub, PDFs, videos, and more into a single knowledge asset, with conflict detection and pairwise synthesis across sources.
Search index inside the skill — query before you read (opt-in)
skill-seekers create <source> --index adds a stdlib-only scripts/search.py and a SQLite FTS5 index over the generated references, so an agent can find the right file#anchor before reading a large Markdown file wholesale. Off by default; the Markdown is untouched.
Video extraction — transcripts, frames, on-screen code
YouTube, Vimeo, and local files. Three-tier transcript fallback (subtitles → YouTube transcript API → local Whisper), plus optional visual extraction that OCRs on-screen code from sampled frames.
Quality, sync & scale
Quality scoring with a gate (skill-seekers quality output/react/ --threshold 7) or JSON stdout (skill-seekers quality output/react/ --json), provisional English readability metrics (informational — they never affect the score), doc-change detection with scheduled re-scrapes and notifications, streaming ingestion for very large doc sets, and incremental updates.
🔌 MCP Integration (40 tools)
Skill Seekers ships an MCP server for Claude Code, Cursor, Windsurf, VS Code + Cline, and IntelliJ IDEA.
# stdio mode (Claude Code, VS Code + Cline) python -m skill_seekers.mcp.server_fastmcp # HTTP mode (Cursor, Windsurf, IntelliJ) python -m skill_seekers.mcp.server_fastmcp --transport http --port 8765
Then just ask your assistant: "Package and upload the React skill."
→ MCP Setup · MCP Reference · HTTP Transport
🤖 Installing to AI agents
Skills install automatically into 19 AI coding agents:
skill-seekers install-agent output/react/ --agent cursor
skill-seekers install-agent output/react/ --agent all # every detected agent
skill-seekers install-agent output/react/ --agent cursor --dry-run| Agent | Path | Scope |
|---|---|---|
| Claude Code | ~/.claude/skills/ |
Global |
| Cursor | .cursor/skills/ |
Project |
| VS Code / Copilot | .github/skills/ |
Project |
| Amp | ~/.amp/skills/ |
Global |
| Goose | ~/.config/goose/skills/ |
Global |
| OpenCode | ~/.opencode/skills/ |
Global |
| Letta | ~/.letta/skills/ |
Global |
| Aide | ~/.aide/skills/ |
Global |
| Windsurf | ~/.windsurf/skills/ |
Global |
| Neovate | ~/.neovate/skills/ |
Global |
| Roo Code | .roo/skills/ |
Project |
| Cline | .cline/skills/ |
Project |
| Aider | ~/.aider/skills/ |
Global |
| Bolt | .bolt/skills/ |
Project |
| Kilo Code | .kilo/skills/ |
Project |
| Continue | ~/.continue/skills/ |
Global |
| Kimi Code | ~/.kimi/skills/ |
Global |
| IBM Bob | .bob/skills/ |
Project |
Uploading to Claude
export ANTHROPIC_API_KEY=sk-ant-... skill-seekers package output/react/ --upload # package + upload skill-seekers upload output/react.zip # upload an existing zip
No API key? Package it and upload output/react.zip manually at claude.ai/skills.
⚙️ How it works
graph LR
A[Documentation Website] --> B[Skill Seekers]
B --> C[Scraper]
B --> D[AI Enhancement]
B --> E[Packager]
C --> F[Organized References]
D --> F
F --> E
E --> G[AI Skill .zip]
G --> H[Upload to AI Platform]
- Scrape — extract every page (checking
llms.txtfirst) - Categorize — organize content into topics (API, guides, tutorials, …)
- Enhance — AI writes a comprehensive
SKILL.mdwith examples - Package — bundle into a platform-ready artifact
- Upload — ship it to your AI platform (optional)
Architecture
8 core modules + 5 utility modules (~200 classes):
| Module | Purpose |
|---|---|
| CLICore | Git-style command dispatcher, source auto-detection |
| Scrapers | 18 source-type extractors on a shared build layer |
| Adaptors | 22 output platform formats behind one SkillAdaptor ABC |
| Analysis | C3.x codebase pipeline, 10 GoF pattern detectors |
| Enhancement | AI improvement via a single AgentClient transport |
| Packaging | Package, upload, and install skills |
| MCP | FastMCP server (40 tools, 10 tool modules) |
| Sync | Doc change detection and notification |
→ UML Architecture · API Reference · Skill Architecture
🆕 New in v3.9.0
- HTML parser fallback for broken markup (#96) — severely malformed pages no longer scrape as empty; well-formed pages are byte-identical.
- Transient-failure retries — the doc scraper (#97) and MCP
fetch_config(#92) now retry connection blips and 5xx with backoff; 4xx still fails fast. - Whisper transcription fallback (#420) — local videos without subtitles finally get a real transcript.
- MiniMax image OCR + registry-driven multimodal providers (#423) — providers declare their wire protocol and image capability; China-issued keys work against the right endpoint.
- Token-lean GitHub issue defaults (#169) — GitHub skills no longer bundle full closed-issue history by default.
- Env-driven CORS across all three servers (#422, #424) — no more wildcard origins with credentials.
Full history: CHANGELOG.md
📈 Performance
| Documentation size | Time | Output |
|---|---|---|
| Small (< 100 pages) | 5–10 min | ~2 MB |
| Medium (100–500 pages) | 15–30 min | ~10 MB |
| Large (500–2,000 pages) | 30–60 min | ~40 MB |
| Huge (10K–40K+ pages) | Use stream |
See Large Documentation |
🐛 Troubleshooting
skill-seekers doctor # diagnose installation & environment skill-seekers doctor --json # machine-readable diagnostics for CI and agents skill-seekers sync-config # detect config drift
Common issues and fixes: Troubleshooting Guide · TROUBLESHOOTING.md
🤝 Contributing
Contributions are welcome — see CONTRIBUTING.md.
- 📋 Development Roadmap & Tasks — pick any task
- 💬 Discussions — questions and ideas
- 🐛 Issues — bugs and feature requests
📝 License
MIT — see LICENSE.
🔒 Security
🌐 Ecosystem
Skill Seekers is a multi-repo project:
| Repository | Description | Links |
|---|---|---|
| Skill_Seekers | Core CLI & MCP server (this repo) | PyPI |
| skillseekersweb | Website & documentation | Live |
| skill-seekers-configs | Community config repository | |
| skill-seekers-action | GitHub Action for CI/CD | |
| skill-seekers-plugin | Claude Code plugin | |
| homebrew-skill-seekers | Homebrew tap for macOS |
Want to contribute? The website and configs repos are great starting points for new contributors!