Builds verified, citation-checked knowledge packages an AI agent can load — so, on a given topic, it answers from real, checked sources instead of guessing.
pip install kp-build
every citation checked live — or dropped
$0 — verification is free public APIs
Python 3.10+
Ask an AI assistant about something that matters — a health question, recent research, a decision — and it sounds completely confident. Sometimes it's completely wrong. It will even cite a study that doesn't exist.
kp-build fixes that for a given topic. It produces a knowledge package: a small, self-contained folder of the real literature on a narrow subject, that any agent loads to inherit the field at once.
A reusable, machine-checkable asset — not a one-shot answer. Verified once; shared; reused.
An everyday question, and a good test — the kind of thing people ask AI all the time, where being wrong matters and the internet is full of hype.
We asked a model to review the sleep evidence and cite studies. It cited
10.5665/sleep.6072 — confidently. That DOI resolves to nothing; it was invented. Nine of
its ten cites were real, though — which is the trap: you can't tell which one is invented without
checking. It also covered only 6 of the 23 real studies.
So we built a verified package on it. Every claim is backed by a real, checked study, and its core is a five-way evidence-vs-hype map:
A model with the package vs. one without, on a held-out question — scored on citation precision and how much of the evidence it covered.
The model was otherwise competent here — base precision was 0.90. It just invented one citation you couldn't have spotted, and covered only 6 of 23 studies. With the package: 0 fabrications, 17 of 23. (kp-build's cheap pre-screen — a single unaided answer — even rated the model "fine" on sleep; the recall-aware falsify caught the gap it missed. And when a package doesn't help, the tool says so.)
The package is public — any agent can install it:
kpm add github:Treibs/kp-sleep-insomnia-evidence#v0.1.0
A research skill (you + AI subagents) gathers and drafts. A deterministic engine then verifies, assembles, and scores — it never trusts the drafting; it checks everything.
A paper is kept only when an explicit arXiv id or DOI resolves and its real title matches. A "real id, wrong paper" mislabel is rejected; the gate never rescues a bad id with a fuzzy search.
Where the source text is reachable, each claim's quoted passage is machine-confirmed to actually appear in the paper — not just asserted.
A probe first asks whether the model is even weak on the topic. After building, falsify measures whether the package beats an unaided agent — and says so plainly when it doesn't.
$ kp-build build -i sleep.research.json -o pkg verifying 23 citation(s) against Crossref / OpenAlex ... citations: 23/23 verified validation: OK 45 claims · 7 open problems · 5 debates $ kp-build falsify pkg --question "improving sleep" --base base.txt --kp pkg.txt base: cited a study that doesn't exist · covered 6/23 pkg: 0 fabrications · covered 17/23 VERDICT: KP HELPS — f1 0.40 → 0.85
Together they show what the checks discriminate — and that kp-build works beyond arXiv (journal papers verified by DOI).
| package | topic | did the package help? |
|---|---|---|
| agent-memory | LLM agent memory — AI frontier | yes — base fabricated/mislabeled 10 of 16 cites; f1 0.49 → 1.00 |
| coding-agents | autonomous AI coding agents | yes — base fabricated 14 of 25 cites; f1 0.48 → 1.00 |
| sleep-insomnia-evidence | everyday health — evidence vs hype | yes — base fabricated a study + missed ¾ of the evidence; f1 0.40 → 0.85 |
| discrete-diffusion-llms | recent ML (model fabricates) | yes — fixes mislabeled cites + coverage; f1 0.37 → 0.91 |
| speculative-decoding-llms | ML the model knows cold | coverage only — precision was already perfect |
| rubric-based-rl-nonverifiable | a 2026-emergent ML topic | yes — coverage 0.07 → 1.00 |
| glp1-incretin-obesity | biomedical (non-arXiv, by DOI) | on coverage — recall 0.26 → 0.95 |
Scores are 0–1, higher is better; f1 combines citation precision (cites that are real and correctly labeled) with coverage (how much of the verified paper set was found).
# install the engine $ pip install kp-build # in Claude Code, with the /kp-build skill: /kp-build the evidence on <your everyday or research topic>
Python, two small dependencies. Verification uses free public APIs — no keys, no cost.