jatechan/ai-data-extractor

Extract your own local chat history from AI coding assistants into a single, normalized JSONL format - for fine-tuning, personal analytics, or just backing up years of conversations before an app's local database gets cleared.

Auto-discovers and extracts complete conversation history, including:

• User messages & assistant responses
• Code context (file paths, selections, snippets)
• Code diffs / suggested edits, where the tool records them
• Tool calls and their results
• Timestamps, session IDs, project paths, model names - whatever each tool's storage actually contains

Every script searches macOS, Linux, and Windows conventions (~/Library/Application Support, ~/.config, ~/.local/share, %APPDATA%, %LOCALAPPDATA%) automatically - you don't need to tell it which OS you're on.

They're two of the most-used AI coding tools that the original list left out, and both have a genuinely different (and instructive) storage shape:

• Cline (and its fork Roo Code) is the most popular open-source autonomous coding agent extension. It stores raw Anthropic-format message arrays per task, so its extractor doubles as the simplest example of parsing that format if you want to add your own tool later.
• Aider is the most popular terminal-only pair-programming tool, and it's structured completely differently from everything else here: no central database, just a markdown transcript sitting in every project directory. It's included specifically to prove the toolkit generalizes beyond "SQLite or JSONL in one app-data folder."

# No dependencies -- standard library only
python --version   # 3.9+ required, 3.10+ recommended

# Interactive: pick which sources to extract from a numbered menu
python extract.py

# Or drive it directly
python extract.py --all
python extract.py --sources cursor,claude_code,aider
python extract.py --list                    # just show what's installed, don't extract
python extract.py --all --merge             # also write all_conversations.jsonl

# Shorthand for "extract everything"
./extract_all.sh

python extract.py [--all] [--sources ids] [--list] [--output-dir DIR]
                    [--search-path PATH ...] [--merge]

--all                 Extract every supported source, no prompts.
--sources ids         Comma-separated source ids (see table above for names,
                       or run with --list to see them). Skips the menu.
--list                Report what was found for each source without
                       extracting anything -- a fast, safe preview.
--output-dir DIR      Where to write JSONL (default: ./extracted_data)
--search-path PATH    Extra directory to search, on top of the normal OS
                       locations. Repeatable. Mainly useful for:
                         - Aider, which has no fixed app-data folder and
                           needs to know where your projects live
                         - nonstandard install locations for anything else
--merge               After extracting, also concatenate everything into
                       all_conversations.jsonl

Each extractor can also still be run standalone, same as the original toolkit (python -m extractors.cursor from the project root, or python extractors/cursor.py), which is handy while debugging one source.

Every run creates timestamped files under extracted_data/:

extracted_data/
├── claude_code_conversations_20260816_143022.jsonl
├── cursor_conversations_20260816_143022.jsonl
├── aider_conversations_20260816_143022.jsonl
├── cline_conversations_20260816_143022.jsonl
└── ... one file per source you extracted, plus all_conversations.jsonl if --merge

Each line is one JSON conversation:

{
  "messages": [
    {
      "role": "user",
      "content": "How do I fix this TypeScript error?",
      "code_context": [
        {"file": "/Users/you/project/src/index.ts", "code": "const x: string = 123;"}
      ],
      "timestamp": "2026-01-16T14:30:22Z"
    },
    {
      "role": "assistant",
      "content": "The error occurs because you're assigning a number to a string type...",
      "tool_use": [{"name": "edit_file", "input": {"path": "src/index.ts"}}],
      "timestamp": "2026-01-16T14:30:25Z"
    }
  ],
  "source": "cursor-composer",
  "session_id": "c1a2b3...",
  "project_path": "/Users/you/project",
  "name": "TypeScript Type Error Fix",
  "created_at": 1705414222000
}

Fields vary a bit by source (not every tool records code_context, token usage, or project_path) - messages, source, and session_id are the only ones you can always rely on.

1. Detect the OS and build a list of plausible data roots (Application Support, .config, .local/share, %APPDATA%, etc).
2. Search each root for the tool's known folder name(s).
3. Read the storage - JSONL line-by-line, SQLite via a read-only connection (so a running app never blocks us), or JSON trees, depending on the tool.
4. Normalize whatever's found into the messages[] schema above.
5. Write JSONL, one conversation per line, into extracted_data/.

Nothing here ever opens a database for writing, and every reader is wrapped so that one corrupt or locked file can't take down the whole run - you'll get a partial result and move on, not a stack trace.

None of these publish their storage schema, and it has changed multiple times (Cursor alone has gone through at least three shapes: workspace ItemTable chat, inline composer, and split bubbleId composer). The cursor.py extractor implements all three known shapes explicitly. windsurf.py and trae.py instead use a generic heuristic (extractors/common.py::heuristic_extract_chat_from_kv) that scans chat-related keys and walks the parsed JSON looking for objects shaped like a role + text pair. It's honest best-effort, not a documented format - if a future version changes shape and stops matching, that's expected; adjust KEY_HINTS in the relevant file or send a PR.

Aider has no central session store - every project directory gets its own .aider.chat.history.md. By default this toolkit scans your home directory plus a handful of common project-root names (projects, code, dev, repos, workspace, src, Documents) up to 5 directories deep, skipping node_modules, .git, and similar. If your projects live somewhere else, point at them directly:

python extract.py --sources aider --search-path ~/client-work --search-path /mnt/data/repos

Every extractor is a small module with the same two-function interface - copy the simplest one (continue_ext.py is a good template) and fill in:

DISPLAY_NAME = "My Tool"
SOURCE_ID = "my_tool"

def find_installations(extra_paths: list[Path] | None = None) -> list[Path]:
    """Return the directories/files worth scanning."""

def extract(installations: list[Path]) -> list[dict]:
    """Return a list of conversation dicts matching the schema above."""

Then register it in extract.py's REGISTRY list. extractors/common.py has the SQLite/JSON/JSONL readers and the two generic heuristics you'll probably want.

This extracts data from tools running under your own user account. Before sharing or training on it:

1. Scan for secrets:

   pip install detect-secrets --break-system-packages
   detect-secrets scan extracted_data/*.jsonl

2. Review for proprietary code, API keys, and personal file paths - code_context and tool_use fields are the most likely places to find them.
3. Don't commit extracted_data/ to a public repo (it's already in .gitignore). Keep it on encrypted storage if it contains client or proprietary work.

from datasets import load_dataset

dataset = load_dataset("json", data_files="extracted_data/*.jsonl", split="train")
dataset = dataset.filter(lambda x: any(m["role"] == "assistant" for m in x["messages"]))

def format_chat(example):
    return {"text": tokenizer.apply_chat_template(example["messages"], tokenize=False)}

dataset = dataset.map(format_chat)

"No installation found" - the tool either isn't installed, has no chat history yet, or lives somewhere nonstandard. Pass --search-path to point at it directly, or check extractors/<tool>.py's SEARCH_DIRS / APP_DIR_NAMES constant and add your path there.

Cursor/Windsurf database locked - reads are opened mode=ro specifically so a running editor won't block extraction, but if you still see errors, close the app and re-run.

Windsurf/Trae found an installation but 0 conversations - the heuristic key-matching in common.heuristic_extract_chat_from_kv didn't recognize the current storage keys. Run --list to confirm the app dir was found, then inspect state.vscdb's ItemTable/cursorDiskKV keys directly (sqlite3 state.vscdb "SELECT key FROM ItemTable") and add matches to KEY_HINTS.

This toolkit extracts your own data from AI tools installed on your own machine. You're responsible for:

• having the rights to the extracted data,
• handling any sensitive/proprietary information appropriately,
• complying with each tool's Terms of Service,
• scanning for secrets before sharing or training on the output.

MIT - use freely, including for training ML models.