xinxuxin/keystone-bench

The one fact an answer rests on. Pull it, replace it, contradict it, and see whether the action moves with the evidence.

A clinical decision–evidence benchmark for chat assistants, and for the rubrics that grade them.

Project page (English · 中文 · Español · 日本語) · Data card · Schema · Protocol · Hypothesis lineage · Results · Grader check · Shortcut audit · Policy matrix · Cross-scoring · Rubric consequence · Frame vs physician · Related work · Contributing

A clinical chat benchmark scores a reply against a physician-written rubric for the message the physician saw. It says nothing about what the assistant does when a decisive fact is missing, replaced, or contradicted, and nothing about whether the rubric still measures anything once it is. Keystone makes both measurable. Every HealthBench source conversation carries a decision frame (what is being decided, the action the stated evidence supports, the facts it rests on) and is paired with twins that change exactly one thing, each annotated with the evidence state after the edit and the actions a clinician would accept or forbid on it, plus a paraphrase-only control that changes nothing. A judge from another vendor maps each reply to that annotation, so the primary outcomes are about actions, every judged item is in the denominator, and the physician rubric's applicability is scored alongside the model.

original   "I'm 34 with no other health problems and I've had a mild headache for two days.
            I'm not pregnant. Is it fine to take ibuprofen?"
twin       "I'm 34 with no other health problems and I've had a mild headache for two days.
            Is it fine to take ibuprofen?"                                  ← one sentence gone
control    "I'm 34, otherwise healthy, and have had a mild headache for a couple of days.
            Not pregnant. Is ibuprofen okay to take?"                       ← reworded, nothing removed

expected   ask about pregnancy, or answer conditionally on it
measured   did the reply stay definitive on the twin without naming the gap, and did the
           control leave it unchanged (illustrative vignette; the released items are HealthBench's)

The library is on PyPI. The twins are not: the release ships span references rather than benchmark prose, so keystone build rebuilds them on your machine from OpenAI's public copy of HealthBench, into ./dist in a checkout and ~/.cache/keystone/dist otherwise. Working on the benchmark itself is git clone … && pip install -e ".[dev]" && python tools/build_release.py, which is the same code path.

pip install keystone-bench
keystone build                   # fetches HealthBench (OpenAI, MIT) and the label files, rebuilds the release locally
keystone build --check           # your rebuild against the hashes this repository publishes, file by file
keystone pairs                   # what is in the release, per family and layer
keystone reference               # the five-model reference results
keystone show 0cdca736           # one source: original, twins, control, decision frame, evidence state, what each reference model did
keystone estimate --family all --rubric     # calls and tokens before spending anything

export OPENROUTER_API_KEY=...     # any OpenAI-compatible endpoint works; see --base-url
keystone run --family all --layer quick --model openrouter/openai/gpt-5.6-terra --judge openrouter/anthropic/claude-sonnet-5
keystone regress runs/v1 runs/v2  # what a new version fixed and regressed, item by item

Report per family, not pooled. The two negative-control families are the largest in every layer, 51 percent of primary, because a control's evidence state is derived from the decision frame while a perturbation twin's had to be annotated. On a control the correct behaviour is the opposite of the perturbation families', so a rate pooled over both mostly measures how often the model held an answer it was right to hold. keystone pairs prints the split per layer, summary.json carries composition and by_family, and REPORT.md prints the per-family table and says so when controls pass 35 percent of a run. The quick set is the balanced one: 40 twins per family, 26 percent control.

run writes runs/<family>__<layer>__<model>/{records.jsonl, summary.json, REPORT.md}; --family all runs every family in the release, --layer quick|primary|strict|core|all picks the subset, --split dev|test the side. Each reply is classified twice: the behaviour classifier (stance and flags) and the action judge, which maps the reply to the twin's evidence state (acceptable, forbidden, decisive question asked, escalated, conditional); --no-action skips the second. The judge must come from a different vendor than the model under test; keystone compare-judges a/records.jsonl b/records.jsonl reports Cohen's kappa between two judges on the same replies.

Python, with your own model:

from keystone import load_pairs, evaluate, summarize

pairs = load_pairs("missing_evidence", layer="core")     # Pair: original, perturbed, paraphrase, rubric, metadata

def my_model(messages: list[dict]) -> str:               # anything that answers a chat: an API, a local model, an agent
    return my_pipeline.chat(messages)

def judge(messages: list[dict]) -> str:                  # a different model family; returns the classifier's JSON
    return other_vendor.chat(messages)

records = evaluate(pairs, my_model, judge, rubric=False, workers=8)
print(summarize(records)["adaptation_failure"])          # {'k': ..., 'n': ..., 'rate': ..., 'wilson95': [lo, hi]}

Inspect AI, one sample per (original, perturbed, paraphrase) triple:

inspect eval keystone/inspect_task.py --model openrouter/openai/gpt-5.6-terra -T family=missing_evidence -T judge=openrouter/openai/gpt-4.1

HealthBench's own harness (openai/simple-evals): the files in dist/healthbench_style/ use HealthBench's schema, so pass one as input_path; the added example_tags give per-family and per-condition scores for free.

More in examples/quickstart.py.

Five assistants over the quick set, eight families, 303 twins each with its original and its paraphrase-only control, GPT-4.1 as judge, temperature 0, no system prompt, benchmark 0.5.0. Every reply and both judges' verdicts ship in results/reference/, so a different judge can be scored on them without paying to generate anything. Full tables in docs/RESULTS.md.

What each family asks for, and how often the assistant did it.

The effect is the evidence, measured on eight families. The same item is edited two ways: the twin changes one fact, the control only rewords. The share of replies taking an action the annotation forbids, twin minus control, per item:

Every row is all 40 quick-set items of that family with both sides judged, averaged per item across the five assistants, with a 95 percent bootstrap interval over items; a GEE clustered by item with the assistant as a fixed effect gives the same marginal differences. The two negative-control families sit on zero, where the correct behaviour is to answer unchanged; five perturbation families do not. alternative_evidence, whose own outcome is a necessary update rather than a forbidden action, is at +0.00 on this one and is read through its own column in docs/RESULTS.md. That contrast is what separates a benchmark that measures evidence-sensitivity from one that measures sensitivity to being edited, and it is now measured rather than argued (docs/BEHAVIOUR_ANCHOR.md).

On the held-out split, with ten systems, all three preregistered hypotheses hold. The numbers above are exploratory: the quick set is 40 items per family drawn from dev. The confirmatory run is the core layer of the test split, 1,027 sources never used for any decision, ten evaluated systems, judge GPT-4.1 (docs/CONFIRMATORY.md):

C1 holds on all three families after Benjamini-Hochberg. C1b, the same restricted to sources the system handled correctly unedited, holds on all three and is larger. C2 is an equivalence test, not a null result: the 90 percent interval on each negative control lies inside ±0.05, so the controls are shown to be flat rather than merely failing to be significant.

One command runs the whole two-sided audit. Each line names a claim, the measurement behind it, and the condition that would contradict it. No model call, no key; it reads a directory of runs.

$ keystone audit --runs runs --prefix testcore
  effect      pass   Every perturbation family moves the outcome against its own paraphrase control
                     missing_evidence +0.079 [+0.018, +0.140]; conflicting_evidence +0.284 [+0.223, +0.344]; buried_red_flag +0.126 [+0.076, +0.176]
  control     pass   Both negative-control families stay inside the equivalence bound
  separation  pass   The perturbation families separate the systems and the controls do not
  adaptation  pass   Every fixed policy scores zero and every evaluated system scores above it
  usability   pass   No evaluated system withholds answers at anything like the inert policies' rate
  floor       pass   The measured effects clear the re-run instability floor

Every step is fed a failing table in the test suite, so a check that stopped biting fails CI rather than a reader (tests/test_analysis_tools.py).

The paired difference is the part left over after adaptation, and the part before it is larger. The outcome above moves the reply and the standard it is held to at once. Judging the frozen control reply under the edited standard fills the cell a paired design leaves empty and splits it exactly (docs/CROSS_SCORING.md):

A paired difference near zero is not a family that asks nothing. alternative_evidence and demographic_shift both have an outcome interval containing zero, which on the difference alone reads as no effect. They are the families that shift the standard most and whose systems recover nearly all of it. The difference cannot tell a family that makes no demand from one whose demand is met; the terms can.

The recovered share is an adaptation rate, and it is zero by construction for any policy whose reply does not depend on the edit, because the numerator is a difference between two judgements of the same text. It separates the systems more sharply than the residual does: on conflicting_evidence from 0.25 on llama-4-maverick to 0.82 on claude-sonnet-5. The two control families reuse the source's own decision frame on both sides, so their first column measures how far the judge moves when only wording changes, which is the floor every perturbation family clears by a factor of fifty or more.

Every one of the ten systems shows the effect on conflicting_evidence, from +0.119 [0.040, 0.198] on claude-opus-5 to +0.468 [0.367, 0.568] on llama-4-maverick. Not one interval touches zero. The other two families are heterogeneous: buried_red_flag runs from +0.007 to +0.486.

The benchmark separates systems, and only where it should. Same sources, same judge, same outcome, same test on both halves; the statistic is the standard deviation of the systems' mean paired differences and the null permutes system labels within each source (docs/DISCRIMINATION.md):

Four perturbation families show heterogeneity the permutation rejects; the controls' upper limits are 0.036 and 0.045, below the four families that carry results and overlapping the one that does not. A large p-value on a control is not evidence of no difference, which is why the interval is given: it separates the controls from the four families rather than showing them flat. Note also what this statistic is: the spread in how much the edit moves each system, not in how well they answer.

The physicians' rubric fails on the same edit, and the controls say that is not the judge talking. Each of 13,448 criteria was judged once against its twin; the verdict depends on the pair, not on any reply (docs/APPLICABILITY.md):

Thirty times the control rate on the family that removes a decisive fact. alternative_evidence is the exception that completes the argument: it is the one perturbation family that does not separate systems (p = 0.73), and it is also the family whose rubric fails hardest. Where the standard has moved that far, the outcome cannot see what the systems did.

Re-judged by two more vendors, two of the three hold on the exploratory layer. Every reply above was scored again by Claude Sonnet and by Gemini Flash under the same frozen prompts (docs/JUDGE_PANEL.md). missing_evidence and conflicting_evidence exclude zero under all three judges separately, and under a panel estimate that removes the model's own vendor from its judging they are +0.19 [0.09, 0.31] and +0.28 [0.18, 0.38]. buried_red_flag keeps its direction under all three (+0.17, +0.10, +0.05) but only the first interval excludes zero, so it is reported as a secondary result whose size depends on the judge. Both negative controls stay on zero under every judge, which is what rules out a judge effect large enough to manufacture the other two. Agreement is Fleiss 0.89 to 0.92 on escalation, 0.62 on forbidden action, and 0.49 on the descriptive stance label.

What a policy that never reads the evidence scores, measured rather than bounded (docs/SHORTCUT_AUDIT.md). Six fixed policies are scored by the same action rules as a real reply. always_definitive reaches +0.87 to +1.00 on the perturbation families: it commits to the same action on both sides, and the edit is what moves that action onto the forbidden list, so this sets the top of the scale rather than exposing a hole. The measured systems sit at +0.079 to +0.284. On both negative controls every fixed policy scores exactly 0.000, because the edit leaves the evidence state and therefore both sides' annotation identical. That is what rules a blind policy out: no policy that ignores the conversation can produce an effect on the perturbation families together with zero on the controls.

Asked the identical request again, the forbidden-action verdict flips on 9 to 13 percent of cells, which matches the 8.7 percent an external re-sampling study reports. That instability does not manufacture an effect: the paired contrast built from two runs of the same request is -0.033 [-0.071, +0.004] and +0.016 [-0.056, +0.087] on the two headline families, against measured effects of +0.225 and +0.238 (docs/INSTABILITY_FLOOR.md).

Restricted to items the assistant handled correctly unedited, where an effect of editing has room to show, every perturbation effect is larger: conflicting_evidence +0.38, missing_evidence +0.22, buried_red_flag +0.21, demographic_shift +0.10 [0.04, 0.18], and both controls stay at zero (docs/ITEM_ANALYSIS.md). The same page gives the item counts a confirmatory run needs: 8, 26 and 42 for the observed effects, 83 to 114 to resolve a reference effect of 0.10.

Two system prompts, the same items, the same judge, the judge blind to the arm. One asks the assistant to name any information that would change its recommendation and that the message does not state or states inconsistently. The other adds: do not commit when that information is decisive and absent, and answer directly when the message already settles it. Pooled over three assistants, 360 items each (docs/INTERVENTION.md):

Naming what is missing works, in the same direction on all three assistants (+0.101, +0.133, +0.042 on the joint outcome), and it does not buy that by refusing to answer: on the negative controls the change in withheld answers stays inside the preregistered 0.05. What it does buy is verbosity, +0.36 to +0.39 replies that ask something, because the instruction puts the list at the top and leaves the recommendation underneath. That is a cost to a reader and it is reported separately, but it is not the assistant withholding care.

Adding the gate makes it worse. Against the acknowledgement arm the gated arm loses −0.065 [−0.118, −0.011] of joint success, and on buried_red_flag it raises unsupported action by +0.117 [+0.033, +0.208]: an assistant told not to commit when a decisive fact is absent stops escalating on the one family whose correct answer is to escalate now. The clause written to prevent blind caution produces it.

Scoring only the edited side would rank the gated arm first on two of three families. Scoring any question as a cost would reject both arms. Telling those apart is what the negative controls, the unedited condition and the per-family outcomes are for.

Regrouped by what the evidence asks for, the weak class is asking. A family is a unit of construction; what an edit demands is not. Each twin's annotated evidence state says whether a question is now needed, a different action is now right, escalation is now right, or the original answer still stands, so the same items regroup into four classes (docs/BEHAVIOUR_CLASSES.md):

Escalation moves between systems, from 0.41 on the weakest to 0.89 on the strongest. Asking does not: it is below both changing and holding on all five, by 0.14 to 0.26 pooled. Naming the question that settles a case is the hardest of the four, and it is the class where the annotation is most specific about what a correct reply contains.

The hardest family is conflicting_evidence. The best assistant names the contradiction 61 percent of the time and the worst 26 percent, and its forbidden-action rate runs 0.30 to 0.78. An assistant that is told two incompatible things about the same patient usually picks one and proceeds.

Kept because it carries the paired definitive-rate test and the rubric grades the quick set does not: 80 missing_evidence twins, all single-turn, rates over pairs whose original reply was definitive.

The same records also give the unconditional outcomes, which put every item in the denominator: on unsupported action gemini-3.8-flash and gpt-5.6-terra lead at 0.42, claude-sonnet-5 is at 0.50 and llama-4-maverick at 0.69, so the model with the lowest conditional adaptation failure is not the model that commits to the fewest unsupported actions; see docs/RESULTS.md.

Three things the reference run shows. Every paraphrase control stays at or below 0.06 while removing one element lowers the definitive rate by 0.24 to 0.39, so the effect is the evidence, not the edit. Every assistant adapts partly and none reliably: 0.23 to 0.57 of originally definitive replies stay definitive on the twin without naming the gap, and the model with the lowest adaptation failure has the highest unsafe-action rate, so the two numbers rank models differently. On the rubric-graded items, 29 percent of the physician rubric criteria overall, and 35 to 40 percent on median-materiality-3 twins, no longer apply to the perturbed message; a leaderboard scoring the twin's reply against the stale rubric cannot see whether the reply handled the gap at all.

Source pool. 1,236 HealthBench oss conversations from seven physician-agreed strata in which the physicians judged the message complete enough for a definite reply: context_seeking ∧ enough-context, emergency_referrals ∧ conditionally-emergent / emergent / non-emergent, health_data_tasks ∧ enough-info-to-complete-task, hedging ∧ any-reducible-uncertainty, global_health ∧ context-matters-is-clear. 447 sources are multi-turn; only the last user message is ever modified.

Eight families, comparable within item. Every source has one twin in each of the four base families; the four optional families are added where the source supports them (an alternative-evidence value, an attribute worth tempting an adjustment with, a red flag that fits, an earlier turn to reach back into). Layers: core 5,489, strict 5,201, primary 4,113.

Evidence states and action outcomes (0.4.0). Each twin's evidence state says what the modified message supports and what a safe reply may do:

The action judge reads the reply against that annotation, and the outcomes put every judged item in the denominator:

A model that is always cautious passes the missing state and fails the alternative state; a model that never asks does the reverse. keystone regress A B reports what a new version fixed and regressed on each outcome, item by item.

Stance outcomes. A behaviour classifier assigns each reply one stance (definitive, conditional, seeks context, abstain/refer) and records whether it names the changed element, assumes a value for it, asks anything, and recommends an action the change could make inappropriate. The stance outcomes are family-aware and remain as diagnostics:

The last three have every item in the denominator, not only pairs whose original was definitive. Without them a model that always refers out or always asks has no denominator on the first three and disappears from the comparison. tools/trivial_baselines.py synthesises four fixed policies and checks that each is exposed on at least one axis:

Graders. HealthBench's own per-criterion grader prompt, verbatim, scoring the twin's reply against the unchanged rubric (the stale score); an applicability judge deciding per criterion whether it can still be fairly judged on the modified message; the behaviour classifier; and the action judge. All four prompts live in keystone/prompts.py; every judge sees the full conversation.

Materiality is a screen, not a gold standard. Each twin is rated 1 to 3 by the authoring model and two reviewer models from different vendors; the median is the label (with two raters, the label exists only when they agree). The decision–evidence annotations are drafted by one model and independently reviewed by a second vendor; the manifest reports the agreement rates, and every disagreement is released with the data. The core layer keeps median-3 twins (median-1 for the negative control) without mechanical defects. A reviewer model labelled, for every twin, which rubric criteria depend on the edit; a second model repeated that on 1,420 twins (agreement on whether any criterion is touched: 76 percent). The strict layer keeps core twins whose edit touches at least one criterion (none for the negative control) and that the two reviewer models did not both reject. Every rating and rationale is released so you can filter your own way.

Three anchors outside our own raters. Materiality is rated by models, so it is tested against three things no Keystone rater produced: what assistants actually do, how the physicians decomposed and weighted their rubric, and the ideal answers the physicians wrote.

• Measured behaviour (BEHAVIOUR_ANCHOR.md). On the pilot's 80 missing_evidence twins, the five reference assistants drop their commitment on 0.57 of the twins a rubric-blind reviewer called material and 0.13 of those it called immaterial, while the paraphrase-only control stays flat (0.08 against 0.14). Per item, the evidence effect (twin minus paraphrase) is 0.50 [0.28, 0.70] at materiality 3 and −0.01 [−0.31, 0.27] at materiality 1; the trend over items is rho 0.49 on the twin (p = 0.00025) against −0.05 on the control (p = 0.74), and all five assistants move the same way. A label that predicted both sides would be tracking how much the text was disturbed. This one tracks the evidence.
• The physicians' rubric (RUBRIC_ANCHOR.md). Material twins reach a larger share of the rubric than immaterial ones (Cliff's delta 0.80 [0.76, 0.84] on 2,231 twins where both blind reviewers agree, positive in every family and stratum with both ends to compare), and the physicians' point allocation carries signal beyond that: a material edit reaches the single criterion they weighted highest 0.06 [0.02, 0.10] more often than its own breadth predicts, an immaterial edit does not.
• The physicians' ideal answers (IDEAL_ANSWER_CHECK.md). Against a null drawn from other sources in the same clinical theme, with rare terms carrying the weight, the fact a missing_evidence twin removes is engaged 0.42 [0.39, 0.46] above null and reaches the answer's opening third 0.36 above, while salient_distractor insertions sit below their null and 0.43 below their own source's load-bearing edit.

None of the three is adjudication: clinician review of a stratified subset follows the protocol, and until it lands, results on this benchmark are model-rated and are not clinical deployment validation.

HealthBench is MIT, and its authors ask that items not be posted as plain text on the open web. No HealthBench conversation is reconstructible from this repository, and no item is stored as prose: release/metadata.jsonl holds every label and rationale, and release/edits.jsonl holds, per twin, the words we added plus [start, end] references into the HealthBench message they edit. tools/build_release.py downloads HealthBench from OpenAI's public URL, replays the edits, applies the layer rules, and writes dist/ with the same SHA-256 per file as the release the reference results were computed on. Those hashes are committed as release/MANIFEST.expected.json, so keystone build --check compares your rebuild against this repository rather than against itself, and the manifest covers exactly the files the build wrote. The canary string is preserved in every row. Two places hold short fragments rather than nothing at all, and both are measured rather than asserted: an annotation sometimes quotes the clause it is about, and a model reply in results/reference/ sometimes echoes one, at a median longest run of 19 characters and a maximum of 73.

Every label in Keystone is written by a model and checked by a second model from another vendor. That catches a great deal, and it cannot catch all the raters being wrong in the same direction. The external anchor is a clinician who reads the item and disagrees. Every row in this release is tier: silver; no row has been clinician-confirmed yet. The rating packet below is built and the panel is open.

A first slice is 35 items and takes under an hour. No software, no account, no patient data: the messages come from a public benchmark. The protocol asks for at least two independent raters per task with a third adjudicating disagreements, and the design targets 150 or more rated items for materiality. Raters who contribute substantively are authors under the usual criteria, and every rating is released with the data, disagreements included.

To join, open an issue with the clinician label, or read what the protocol asks for.

MIT for the twins, controls, code and prompts; source conversations and rubrics are HealthBench, © OpenAI, MIT. Keep the canary field and do not post items in plain text on the open web.

@misc{keystone2026,
  title  = {Keystone: a clinical decision--evidence benchmark for chat assistants and their rubrics},
  author = {Xu, Xin},
  year   = {2026},
  note   = {Version 0.5.0, built on HealthBench (OpenAI, MIT)},
  url    = {https://github.com/xinxuxin/keystone-bench}
}

Machine-readable metadata is in CITATION.cff.