AutoHarness
Self-Learning Skills for Claude Code
autoharness is a self-learning skill layer for Claude Code. It learns skills from your real sessions, merges same-scenario ones instead of stacking near-duplicates, updates them in use, and prunes any that stop getting used — so the layer stays clean on its own, touching only the skills it wrote itself.
Same model, different harness — 42% → 78% on CORE-Bench (HAL). The harness does much of the work (swyx's Big Model vs Big Harness), yet it's still rebuilt by hand every model generation. autoharness bets one slice of it — the skill layer — can maintain itself.
| Learns from real work | Each episode is distilled into a skill from the session you were already having — no separate data-collection or replay loop. It fires on its own once a session has done enough work; /learn distills on demand when you want a lesson kept now. |
| Groups, doesn't just pile up | A new episode doesn't always add a skill — the reflector compares it against what's there and folds same-scenario skills into one, so the layer consolidates by category instead of accreting near-duplicates. A fold records which skill absorbed which, so a merge is never mistaken for a death. |
| Keeps its own library in view | Every session opens with a grouped index of the skills it wrote, so recall doesn't depend on the host happening to surface them. The host's native recall is left exactly as it was; the index is added on top. |
| Validated in use, not on a benchmark | A skill survives by being adhered to in later turns (loads over the requests it was available for), not a held-out score. No oracle on the active path, and no tokens spent on a dedicated eval. |
| Only its own skills | Touches only the skills it generated through this plugin — everything else, whether you wrote it or installed it, is left completely alone. |
| Evidence kept for later | Every create/update logs its scenario and decision to a per-skill ledger — the raw material to build a benchmark from real usage if you ever want one. |
Install
Requires Python 3.11+ as the python3 on your PATH — autoharness runs entirely as Python
(zero third-party dependencies); its hooks and MCP server won't fire without it. The hooks resolve
bare python3, so an older interpreter earlier on your PATH (Xcode ships 3.9.6 at
/usr/bin/python3) turns every hook off for the session; autoharness says so on stderr.
Type these in the Claude Code input box.
/plugin marketplace add tigerless-labs/autoharness
/plugin install autoharness@autoharness
Then run /reload-plugins (or restart Claude Code).
Zero config. It now watches your sessions and lands learned skills into .claude/skills/ in the
background. Cadence and lifecycle thresholds are tunable — see Configuration.
Nothing to invoke, but one entry point exists when you want it: /learn distills the session
you're in right now — say it after working something out and the lesson goes through the same
proposal-and-validation chain the background pass uses.
MCP server naming. .claude-plugin/plugin.json registers the server as stage_skill, but agent
definitions reference the fully-qualified name mcp__plugin_autoharness_stage_skill__stage_skill.
This translation is automatic: the plugin runtime constructs the qualified name from the plugin
name (autoharness) and the server key. Outside the plugin context (e.g., registering the server
by hand), the tool would be available as mcp__stage_skill__stage_skill — but the agent allowlists
still reference the plugin-namespaced form. Always install as a plugin to match both names. The
server is declared in the manifest, not in a root .mcp.json, because the plugin root is the repo
root: a .mcp.json there would also be this checkout's project config, which outranks the plugin's
same-command server and leaves the reflector without its tool in sessions inside the repo.
Update
Update and uninstall are shown in their terminal form (claude plugin …); inside Claude Code the same
operations live in the /plugin menu.
Update from a terminal — refresh the catalog by marketplace name, then update the plugin by its
full plugin@marketplace id, then restart:
claude plugin marketplace update autoharness
claude plugin update autoharness@autoharness
Then restart Claude Code to apply — a version bump is a fresh cached copy, not a hot reload.
The refresh is first on purpose: without it, update checks a stale local catalog and may report
already at the latest version when a newer release actually shipped.
Third-party marketplaces have auto-update off by default. To make future releases hands-off,
enable it once: /plugin → Marketplaces → autoharness → Enable auto-update. The
installed copy is cached by the version in plugin.json; a release reaches users only when that
field is bumped.
Uninstall
claude plugin uninstall autoharness@autoharness
claude plugin marketplace remove autoharness
Uninstalling only stops it from running — the skills it landed and its own state live outside the
plugin and stay on disk. To clear those too, delete its state dir (~/.claude/autoharness/ global,
<repo>/.claude/autoharness/ per project) and the self-authored skills under .claude/skills/ (each
carries a self-authored ledger marker, so they're easy to tell from yours). Your own skills are
never touched.
Configuration
Every knob is an AUTOHARNESS_* environment variable with a built-in default — nothing to
configure unless you want to change the pace.
Placement — which layers it manages
| Variable | Default | What it does |
|---|---|---|
AUTOHARNESS_DISABLE_GLOBAL |
0 |
Set to 1 for a project-only deployment. Global create intents are rejected before staging and landing; global request/use/view counters, archiving, orphan sweeps, recall, curator snapshots, and metrics are skipped. Existing global skills stay untouched. Project-layer learning continues normally. |
Cadence — when it learns
| Variable | Default | What it does |
|---|---|---|
AUTOHARNESS_REFLECT_EVERY_N |
50 |
Reflection cadence, counted in tool calls, not turns: every main-session tool call advances a counter, and the turn that pushes it past N ends with a background reflection. A working stretch triggers; a conversation that only talks never does. Lower = learns faster and spawns more child sessions. |
AUTOHARNESS_CONSOLIDATE_EVERY_N |
250 |
Same quantum for the curator, the periodic pass that merges the library as a whole. Held well above the reflection cadence — consolidation is rarer than distillation. |
AUTOHARNESS_DIGEST_EXCHANGES |
20 |
How many exchanges before the episode window are compressed into the reflector's prior-context digest (text + tool names only). Unused by the fork carrier, which replays the real conversation instead. |
AUTOHARNESS_CARRIER |
bundle |
What carries the reflection. bundle hands a redacted window + digest to a fresh subagent. fork resumes and forks the session that just ended, so the reflector reads the real conversation on the parent's warm cache. Stays bundle until the cache-hit measurement is in. |
Recall — what the model sees
| Variable | Default | What it does |
|---|---|---|
AUTOHARNESS_INDEX_SUSPENDED |
0 |
Set to 1 to stop injecting the index entirely. Everything else keeps running — the lifecycle pass, the use/view counters, the last-run summary — so this is the switch for measuring what the index is actually worth, and for anyone unwilling to spend the context on it. |
AUTOHARNESS_INDEX_DESC_MAX_CHARS |
60 |
Per-line description budget in the session-start index. The index is a scan surface, not the full trigger text — raise it for longer lines, at the cost of context on every session. |
AUTOHARNESS_SKILL_DESC_MAX_CHARS |
1024 |
Hard cap on a skill's own description, matching the host's documented limit; a longer one is rejected rather than silently truncated at load. |
AUTOHARNESS_SKILL_BODY_MAX_LINES |
25 |
Altitude cap: a SKILL.md body over this many non-blank lines is rejected as a transcript rather than a rule. Backing detail belongs in the skill's references/. |
Lifecycle — what survives
| Variable | Default | What it does |
|---|---|---|
AUTOHARNESS_MATURITY_PROJECT |
100 |
Probation gate, project layer: after this many requests have arrived in its layer since a skill landed, it faces graduation review. Until then it's recalled as usual but can't be archived. |
AUTOHARNESS_MATURITY_GLOBAL |
300 |
Same gate for the global layer — higher because a global skill loads in every project. |
AUTOHARNESS_CAPACITY_PROJECT |
50 |
Cap on mature skills in the project layer. For graduates, capacity contention is the only death: nothing is archived until the mature pool exceeds this, then the lowest usage rates go first. It does not bound the session-start index: probation members are indexed too and are outside this cap, so the index is a function of the creation rate rather than of the two caps. |
AUTOHARNESS_CAPACITY_GLOBAL |
20 |
Same cap for the global layer — smaller because its blast radius is every project. |
AUTOHARNESS_GRADUATION_SUSPENDED |
0 |
Set to 1 to park graduation review entirely, so nothing is archived for going unused. Meant for when you have reason to doubt the recall surface: archiving on zero use would then be punishing skills for never having been offered. Capacity contention still applies. |
AUTOHARNESS_SNAPSHOT_KEEP |
5 |
How many pre-run snapshots of each skill tree the curator keeps before merging. A merge is the one operation a single atomic rename can't undo. |
Notify — hearing about it as it happens
Off by default. The session-start summary line reports a run one session late, with counts but no names; these push the same run account out as soon as a drain finishes. Fail-open: the notifier runs only after the account is written and the intent queue cleared, so a missing, failing or hung notifier can delay a drain but never fail or replay it.
| Variable | Default | What it does |
|---|---|---|
AUTOHARNESS_NOTIFY |
(unset) | Set to desktop for a native notification per run — osascript on macOS, notify-send on Linux when installed (a silent no-op otherwise). Names what landed and what was rejected, e.g. create foo, patch bar · rejected: baz. On macOS the notification is attributed to Script Editor; if nothing appears, allow it under System Settings → Notifications. |
AUTOHARNESS_NOTIFY_CMD |
(unset) | A command (split like a shell argv, never run through a shell) that receives the run record JSON — the same shape as runs/<run-id>.json — on stdin, with the one-line summary in AUTOHARNESS_NOTIFY_SUMMARY. The hook for Slack, webhooks, or anything else. Skill names are redacted first, and the command runs with the child-session guard set, so a notifier that launches claude isn't itself captured. Works alongside desktop. |
AUTOHARNESS_NOTIFY_TIMEOUT_S |
5 |
Whole seconds, minimum 1, per channel: how long a notifier may hold up a drain. The /learn drain runs inside the host's Stop hook, so this is also the most it can delay a turn. |
Set them in the environment Claude Code launches with — either the shell
(export AUTOHARNESS_REFLECT_EVERY_N=10) or the env map in .claude/settings.json:
{ "env": { "AUTOHARNESS_REFLECT_EVERY_N": "10" } }Hooks read the environment on every event, so a change applies from the next session. The defaults are deliberate placeholders pending empirical calibration; byte caps on captured windows and staged files are fixed constants, not env knobs.
How it works
A learning pipeline runs beside the host. Skills are plain native files, recalled by the host's own name-and-description mechanism as if a human had written them — that path is left untouched. On top of it, each session opens with an index of the skills autoharness wrote, so whether its own library gets offered is a property of this plugin rather than a hope about host behavior.
Diagram source: docs/assets/pipeline.mmd — re-render to pipeline.svg after editing.
| Component | Role |
|---|---|
| CAP · capture | Hook-driven dumb pipe: grabs each turn (user input, agent output, tool I/O), redacts at egress, points back at the host log instead of copying it. It also holds the trigger, which is deterministic and counts one thing — tool calls. The turn that crosses the threshold ends with a reflection; nothing about the content is judged here. |
| REF · reflect | Reads the episode, compares it against the existing skill index, and decides add / merge / patch / drop a support file / delete — emitting an intent (body, delta, or path, plus reason and evidence). Where a new lesson contradicts an older skill, it must rewrite the stale one in the same run rather than leave the library arguing with itself. Proposes only; it has no write tools, and a fork carrier's inherited ones are denied at the hook. |
| promoter · validate·store | The only writer. Lints the intent in memory (safety, structure, ledger, completeness, self-authored-only) and on pass does an atomic rename into the live skill directory. A new skill's description has to carry its trigger early enough to survive the index's truncation — a cue that lands past the cut leaves the skill as half a sentence on the very surface meant to recall it. A fold must name the skill that absorbed the deleted one, and the umbrella has to be a live skill autoharness manages — an invented name fails the whole intent rather than losing the content. Each run leaves an account of what landed and what was rejected. |
| IDX · surface | Builds the session-start index: the skills autoharness wrote, grouped by category, one truncated description per line, tagged by layer. Archived and hand-written skills are excluded, an empty library injects nothing, and the previous run's landed/rejected line rides along — so a rejected proposal is visible instead of silent. |
| MNG · lifecycle | Daemon-free: recomputed lazily at session start, once per session. Ranks symbols by usage rate — loads over the requests that arrived since the symbol was created, so the measure is opportunity-relative and a closed laptop doesn't age anyone out (the wall-clock replacement). Three signals are kept apart: a load (the model invoked the skill) is the only thing the rate counts; a view (a session read into the skill's directory) is evidence it had recall value, but not adherence; a patch marks the skill being improved, so a load after one reads as reuse-after-improvement. New symbols sit in probation until they've had a fair sample of requests: recalled as usual, but neither counted against the cap nor evictable. At maturity, graduation review archives only a symbol that was never loaded and never viewed — no evidence of use is not the same as evidence of no use. For graduates, capacity contention is the only death — nothing is archived until a layer's mature pool exceeds its cap, then the lowest rates go first. Archives, never deletes: an archived symbol is a directory moved out of recall, and moving it back revives it. |
| curator · consolidate | The rarer whole-library pass: reads the library as one thing and folds near-duplicates under umbrellas, which is the judgment a single episode can't make. Snapshots both skill trees before it starts, since a merge is the one operation an atomic rename can't undo. |
| LED · ledger | Per-symbol append-only sidecar: why each symbol was born or changed, with evidence and a reflection watermark. Kept out of the skill body so recall stays clean. |
Walkthrough: watching it learn
Everything autoharness does lands on disk as plain files — a demo is just opening them in the right order. For a fast-paced run, speed up the loop first (see Configuration):
{ "env": { "AUTOHARNESS_REFLECT_EVERY_N": "3",
"AUTOHARNESS_MATURITY_PROJECT": "5",
"AUTOHARNESS_CAPACITY_PROJECT": "2" } }1 · The pipeline running. Work a few normal turns on anything non-trivial (debug something, figure out a workflow). Once a turn pushes the tool-call count past N, a background reflection fires as that turn ends — nothing blocks your session. Its bookkeeping is visible in the state dir:
ls .claude/autoharness/ # per project — ~/.claude/autoharness/ for the global layer
requests # layer request counter (MNG's denominator)
session-<id> # tool calls counted toward the next reflection
offset-<id> # byte watermark: where the last captured window ended
intents/ # queued skill proposals awaiting the promoter
runs/<run-id>.json # what that run proposed, landed, and rejected — with reasons
last_run.json # the summary line awaiting the next session start
snapshots/ # skill-tree tarballs the curator takes before merging
2 · A skill is born. After a reflection lands, a new folder appears under .claude/skills/
(project) or ~/.claude/skills/ (global — for techniques that aren't repo-specific). Use ls -la:
the interesting files are hidden.
.claude/skills/<name>/
SKILL.md # the skill itself — plain native format, nothing proprietary
.ledger.jsonl # LED: why it was born / changed (append-only)
.sidecar.json # lifecycle counters MNG reads
references/evidence-*.md # the transcript slice that justified each ledger entry
scripts/ templates/ ... # optional support files the reflector attached
3 · LED — the paper trail. cat .ledger.jsonl — one JSON line per lifecycle event:
{"action": "create", "reason": "User asked about the correct command to update a plugin ...", "evidence": "references/evidence-21cd22cc.md"}
{"action": "patch", "reason": "User discovered /reload-plugins is required in-session ...", "evidence": "references/evidence-1a4ec51d.md"}action + reason + evidence — and the evidence file is a real, redacted slice of the session
that taught it, materialized by the promoter (content-addressed, so the model never names files).
This is the "evidence kept for later" from the table above.
4 · An update, not a duplicate. Hit the same scenario again with a correction ("that's missing
a step") and let the next reflection run. The skill layer does not grow a near-duplicate:
the existing skill's SKILL.md changes and its ledger appends a patch/update line — the
two-line ledger above is a real example. git diff on a project-layer skill shows the edit.
5 · The next session opens knowing. Start a new session and the first thing it receives is a grouped index of these skills — plus a one-line report of the last run, so a proposal that got rejected says so instead of vanishing. The host's own recall still runs untouched underneath; the index only makes sure the library is in front of the model either way.
6 · Use is counted three ways. cat .sidecar.json: use ticks when the model actually loads
the skill and is the only thing the survival rate counts; view ticks when a session reads into
the skill's directory — recall value, but not adherence; patch ticks when the promoter lands an
improvement, so a use after one is reuse-after-improvement. Keeping them apart is what stops a
skill that was merely browsed from looking like one that was followed.
7 · Retirement is an archive, not a delete. Two paths out, both a folder move to
.claude/skills/.archive/<name>/ — ledger, evidence and all, out of recall. Graduation review
archives a skill only if its whole probation passed with no use and no view; after graduation,
once a layer's mature pool exceeds capacity the lowest-usage-rate skills go. Moving the folder
back revives it, history intact. With the shrunk knobs above this fires within one session; at
defaults it takes hundreds of requests. A skill merged into another is archived the same way, but
its ledger names the umbrella that absorbed it — a fold and a pruning stay distinguishable.
8 · Yours are never touched. Every autoharness-authored skill carries the ledger marker; anything without it — skills you wrote or installed — is invisible to the promoter and MNG.
How it compares
A self-learning skill layer can be validated against a held-out benchmark, or against its own use. autoharness takes the second — cheaper, and it works on a live host doing open-ended work where no benchmark exists.
| Grow unbounded | Offline-gated self-edit (Self-Harness) |
Timer + daemon (hermes-agent) |
autoharness | |
|---|---|---|---|---|
| Bounds the skill layer | No | Yes | Yes | Yes |
| Validation signal | None | Held-out benchmark score | Wall-clock inactivity | Adherence in use |
| What starts a learning pass | — | An offline batch | Idle time and elapsed days | Work done in the session |
| Puts its own library in front of the model | No | No | Yes | Yes |
| Needs a benchmark / oracle | No | Yes | No | No |
| Needs a resident daemon | No | No | Yes | No |
Acknowledgements
NousResearch/hermes-agent — studying its auto-skill-creation and memory-consolidation design helped sharpen autoharness's adherence-based, daemon-free take.
AI Coding Tools (Tigerless Labs Workflow 2026) — a third-party walkthrough by the AI Insider channel; not ours, and not reviewed by us.
Built by Tigerless Labs.
License
© Tigerless · tigerless.ai · tigerless.com