LLM wiki vs. RAG: what the research says about a memory for discovery

On LLM wiki vs. RAG for customer research, what eight recent papers say about the design of a memory for what we have learned from customers, and what we still have to test.

Niklas Luhmann tended his Zettelkasten by hand for decades: one thought per card, every card linked, the interpretation done when the card was written. The pattern is older than the tools. What is new is who does the filing.

A fun fact first, and a bit meta: I wrote this digest with the tools it describes. Claude Code runs my regular content research and files it into our own internal research wiki, where every claim points to the paper it comes from. The same core papers also sit in a Gemini Notebook, the tool formerly known as NotebookLM, and on walks I listen to the podcast it generates. One example from that podcast stuck with me, a pharma research team and a 200-page clinical study, and I couldn't find it anywhere in our synthesis. So I asked Claude why. The answer: the two podcast hosts had made it up.

Checking the rest of the episode against the wiki turned up two subtler slips, both about a paper on self-testing wikis that I come back to at the end. The hosts said half of the critical facts get lost when a wiki is compiled. The paper says half of the answers failed completely, which is not the same thing. And they said the agent writes its own test questions, while the researcher took them from an existing benchmark. Not quite a fair fight, since a podcast is supposed to tell stories, but I'll take it: 1–0 for the LLM wiki over a RAG-style tool working from the same sources.

I'll keep using the podcast, because nothing gets me into a new topic faster. When precision matters, I rely on the wiki. That precision doesn't come with the wiki pattern itself, as the first paper below shows. It comes from how we set ours up: every claim points to the paper and page it comes from, and for this article a second agent checked each one against the PDF.

The wiki idea goes back to Andrej Karpathy, who published his LLM wiki gist in April 2026 as an "idea file". It reads like a recipe for a personal wiki, but one of its examples is a team wiki fed by Slack threads, meeting transcripts, project documents and customer calls.

That example is the one we care about. Discovery done with AI, reaching real customers, interviewing and testing them, synthesizing what comes back, needs a maintained memory of its results and a path from there into assumptions, tests and decisions. Without that memory, every synthesis, pivot and decision starts from scratch.

How we got from a simple RAG chatbot in 2023 to an LLM wiki for a research team, I described in What should a user research agent do?, and the basics of the pattern in Everyone can become a knowledge builder. The difference is when the thinking happens. RAG does it every time someone asks, pulling snippets and making sense of them on the spot. A wiki interprets once, when the document comes in, and writes the result down. Since Karpathy's weekend project went viral (GitHub shows 5,000+ stars and 5,000+ forks), research has been catching up. From a reading list of more than 200 papers I ranked eight by what they teach about a wiki that collects discovery insights, keeps them true and serves them to synthesis, pivots and decisions. Each paper gets three parts: what it means for a research wiki, what you'd have to build to test it, and what we've seen in our own practice.

1. Compiling helps synthesis, but not cheaply

Single-Round Vector RAG vs an LLM-Compiled Wiki: A Preregistered Comparison on a Small Multi-Domain Research Corpus, Theodore O. Cochran (AI for Altruism), preregistered preprint, third version from October 2026. He built an LLM-compiled wiki and a RAG system that retrieves once per question over the same 24 research papers, and put 13 questions to both.

For a research wiki: compiling pays off where a question has to connect several interviews, and it costs real money. When a question needed several sources connected, the wiki linked them clearly better than single-round RAG, but its answers weren't clearly better structured, and it used about 21 times as many tokens per question. Better citations aren't part of the deal: the first two versions of the paper claimed them, and the third withdrew the claim after finding a bug in the scoring script. A link from a wiki page to its source doesn't prove that each sentence on the page is backed by it.

What it takes to test it: a stronger RAG baseline on your own data, because in an exploratory follow-up a RAG variant that splits a question into sub-questions closed most of the wiki's lead in linking sources at a fraction of the wiki's tokens, so the comparison worth running is your wiki against decomposition RAG, or a hybrid. And traceability down to the single claim, built into the wiki rather than expected from it. The study is small: 13 questions, only four of them synthesis questions, scored by LLM judges.

From our practice: we run the expensive configuration on purpose, an agent reading pages without a vector index, and we don't let the model write its own citations. Cochran's scoring bug has a cousin at ours: our LLM judges drifted on an unchanged wiki between two runs weeks apart, so we now score in pairs on the same day. The comparison against a stronger RAG baseline is the one test we still owe ourselves.

2. Interpret once, when the interview comes in

RAG Deserves an Index: Why Ingest-Time Compilation Beats Query-Time Interpretation, Kyle Wild, Yusuke Takahashi and Asako Uraki (Endgame Labs, Musashino University), position paper, August 2026. They compared answering from raw transcript snippets with answering from claims extracted once per document, on a corpus of broadcast interviews.

For a research wiki: if every interview is broken into single claims with verbatim quote and speaker as soon as it comes in, questions get answered more accurately and with far less reading per question than by searching raw transcript snippets, after a one-off extraction cost per interview. In the test, answers from the claims were right in 85 of 100 cases, against 73 for plain snippet search, while reading a fraction of the text. Only a much more elaborate search setup matched it, and it read many times as much.

What it takes to test it: one study is enough to start: break its interviews into quoted claims as they come in, check each quote against the transcript when the claim is written, drop claims whose quote can't be found, and compare the answers with your current setup. Discovery data is messier than broadcast interviews, the paper's evidence is preliminary, and the advantage did not automatically hold once an agent, rather than a fixed pipeline, did the reading.

From our practice: an interview becomes quoted claims the day it comes in, with the transcript one link away behind the reader's own permissions. The quote check earns its keep: our extractor has quietly tidied quotes, and we reset them to the wording on the recording, which is exactly the error class the paper's gate throws out.

3. Agent memory: a pivot is a retraction, not an overwrite

TEPA: Revoking Stale Memories for Conflict-Robust Language Agents, Yan Zhou, Yue Ouyang, Kaiyang Zheng (Changsha University of Science and Technology) and Suncheng Xiang (Shanghai Jiao Tong University), August 2026 preprint. The paper is about the memory of AI agents and what happens when facts they stored stop being true.

For a research wiki: a wiki that only ever adds findings gets worse the moment a pivot reverses one of them, in the authors' simulation even worse than no memory at all, because an invalidated insight is still on topic and search keeps finding it unless a status says otherwise. What helped was a status on every entry: valid entries are used, revoked ones move to an archive and stay traceable.

What it takes to test it: a status on every insight, and a rule for who sets it. A contradicting interview is a valid finding. That is different from a finding that a pivot has made invalid, and pivots, small and large, happen all the time. Our assumption: each insight needs a status that search respects, and an agent can set it, as long as a recorded decision triggers it, not the contradicting interview itself. The evidence is a simulation: TEPA's headline tests ran without a language model, relied on a fixed list of which facts can contradict each other, and on a test with very long documents it did worse than add-only memory.

From our practice: a check surfaces contradictions and leaves the decision to people. In my own vault, outdated documents carry a superseded mark since the day two agents working in parallel decided the same question in opposite directions, because the real decision existed only in a call. The mark works because a decision log sits next to it, and a status per insight, set when a decision is recorded, is what we build next in the research wiki.

4. Entity pages help, theme trees hide evidence

Follow the Entities: A Corpus Map for Agentic Search, Soyeong Jeong, Sujay Kumar Jauhar, Sung Ju Hwang and Andrew Joohun Nam (KAIST and Microsoft), September 2026 preprint. The researchers left the raw documents untouched and added one page per important entity, meaning the things the documents are about, with facts linked to their sources.

For a research wiki: pages for the things your research is about, with the raw documents one link away, are the way in, and a tree of themes as the main entrance hides evidence. In one benchmark of company documents, an agent working with the entity pages found 76 percent of the relevant documents instead of 62 percent, and read less than half as much. Sorting the same material into a tree of themes made things worse: in one case the agent never opened the folder that held the evidence and answered that the document didn't exist. That is the failure a research wiki has to avoid: evidence that exists, filed under a theme nobody opens, and an answer that says there is nothing on this.

What it takes to test it: one page per study, feature, market, segment and experiment, each fact on it tagged with its source, and a keyword search that points the agent to likely pages first, because the token savings in the paper depend on that. Assumptions are trickier, because tying a fact to an assumption is itself interpretation. The evidence comes with a condition: a loosely specified "LLM wiki" did no better than searching the raw documents in this study, so the discipline of the structure matters more than the label.

From our practice: our map is a set of typed entity pages with the raw transcript one link away, and deliberately no page about any individual participant. Defining the entity types ourselves instead of taking the model's defaults was a lesson from an early experiment, and this paper puts a number on it. What we haven't built is the assumption as an entity: ours still live on the study page.

5. Retracting a source needs receipts per sentence

Keep Your Receipts: Trustworthy Cleanup of Agent-Built Knowledge Graphs, Vishisht Choudhary (Technical University of Munich), October 2026, a NeurIPS workshop preprint according to the author. He had an LLM build a knowledge graph, a network of facts, links and summaries, from medical research abstracts, and then took dozens of the abstracts back out.

For a research wiki: a retraction travels only as far as your receipts reach. Removing the abstracts was harder than it looks, because their traces sat in derived links and summaries. A simulated version of the usual cleanup, which deletes only the facts taken directly from the source, barely reduced how often searches still surfaced the retracted material, and rebuilding from scratch changed far more than the retraction itself. What worked were receipts, notes the model writes with each claim about which sources support it, but only per sentence: receipts for a whole summary held in only about a third of cases in the main test. An insight page that only lists its sources at the bottom cannot tell you which sentence a withdrawn interview takes with it.

What it takes to test it: a source reference on every insight sentence, not on the page, and a retraction procedure that follows those references instead of rebuilding. In discovery, retraction is routine: an unusable session, a withdrawn consent, a misattributed quote, so the next one is your test. The receipts don't depend on the medical topic, which was just the test material, but the evidence comes from small samples on this one graph, checked by the same model that built it.

From our practice: an insight sentence cites the facts that carry it, so a retraction knows where to go. When a batch of calls had to be transcribed again, we re-anchored the existing facts in the new transcripts by searching for their quotes instead of distilling everything anew, which is the move Choudhary's numbers favor. The cheaper move is still the gate at the door, so unusable sessions never get in.

6. Separate the librarian from the writer

FS-Researcher: Test-Time Scaling for Long-Horizon Research Tasks with File-System-Based Agents (arXiv), Chiwei Zhu, Benfeng Xu and colleagues (USTC and Metastone Technology), ACL 2026, the only paper here with main-conference peer review. It splits research into two roles: one agent, the librarian, collects sources and writes cited notes into a folder of files, and a second one, the writer, has no web access and builds the report from those notes alone.

For a research wiki: observe first, interpret later, with the two jobs done by two agents that share nothing but the files. The split scored higher than the best competing system on a benchmark of 100 PhD-level research tasks, rated by an LLM. One question from their agents' checklist belongs before every pivot: "Are there any important aspects where you only have 1–2 weak sources?"

What it takes to test it: two agents instead of one, the collector with the sources and the writer without them, and an eval that compares the merged version against the split one on the same questions. The evidence has three soft spots: the lead shrinks when the competing systems use the same underlying AI model, the test that merged both roles into one agent changed other settings too, and the authors' group also built the benchmark.

From our practice: librarian and writer are different agents too. The ones that build the wiki never answer, and the one that answers in Slack reads the wiki and nothing else. In one experiment we gave only the collecting and checking agents the stronger model and kept the writer the same, and the answers got better in every measure we had, on a sample too small to call it proof. This digest was built the same way, scouts and paper agents first, synthesis after.

7. Snippet RAG often fails before the model reads

The Bottleneck Is Retrieval, Not Reading: Diagnosing Evidence-Based Generation with RAG for Long Context Question Answering, Debopriyo Banerjee (Inception42) and five co-authors at MBZUAI, among them Preslav Nakov, October 2026 preprint. They tested snippet-based RAG on news articles where each question needs several facts, and in a second small pilot on scientific papers.

For a research wiki: the questions discovery teams ask, the ones that count or compare across many interviews, like "what are the top pain points?", are exactly where snippet retrieval breaks, before the model even reads. With a single, standard semantic search and no reranking, the top results held all the needed facts for only 19 percent of questions. Snippets also lose track of which document they came from. In the pilot on papers, RAG called answerable questions "unanswerable" about twice as often as a model reading the full documents, which looks harmless in a score but can mislead a go/no-go call.

What it takes to test it: every fact keeps its origin, who said it, in which study, and the eval scores answers and evidence separately, because even with the relevant full articles in context, models gave correct answers far more often than correct answers backed by all the evidence. The evidence rests on one retriever without reranking, small pilots and a judge model nobody calibrated against people.

From our practice: every fact keeps its identity, which study, which role, which market, never a name. The rule that an answer built on thin evidence says so came out of our evals: one question about gaps in the corpus was answered wrong in every run, with product wishes instead of gaps, because the wiki had no place for absence.

8. Automated theming can swallow weak signals

Designing an Auditable LLM-Supported Workflow for Qualitative Thematic Analysis, Nadia Jul Jeldtoft and Tariq Yousef (University of Southern Denmark), August 2026 preprint, the thinnest evidence here. A small local model coded interview passages, wrote a rationale for each code and sorted the codes into themes one at a time.

For a research wiki: a wiki that keeps adding new sessions to existing pages runs the risk this paper shows: a weak signal that would justify a pivot can vanish into an old theme. Compared with one human analysis, the model compressed: 10 themes instead of 19, with the largest theme far bigger than any human one. Two experts who didn't know which was which rated its rationales above the human ones, though on only a handful of passages.

What it takes to test it: a rationale for every theme assignment and a check on theme size, so the compression becomes visible before it costs you a signal. The authors suspect, without having tested it, that sorting codes one at a time into a growing set of themes causes the compression, so this is an untested cause with an untested fix.

From our practice: code does the counting and linking, the model only interprets. Insights are hard. Saeideh Bakhshi set the bar in a Substack note on 20 October: an insight should "change what I believe, what I would do, or what I would investigate next", anything else is an interesting observation. Measured against that, our insight pages still read like theme dossiers, so our next experiment is atomic insight IDs, one per insight rather than one page per theme, each with its own status. The rationale per assignment is what we take from this paper.

Two ideas we haven't tried yet

WiCER: Wiki-memory Compile, Evaluate, Refine (Juan M. Huerta, Zinnia Tech Solutions, May 2026 preprint) tests the wiki against its own sources. After compiling, it asks one test question per source document, and where the answer fails, the lost facts are pulled from the source and pinned for the next compile. That cut completely failed answers from about 55 to 25 percent, with a small local model and test questions taken from the benchmark. Knowledge Compounding (Shuide Wen and Beier Ku, Tsinghua University and University of Oxford, April 2026 preprint) lets the wiki learn from questions: when it lacks an answer, it searches the web and writes the result back as a page, tested on only a handful of questions.

What it takes to test it: a test set made of the questions PMs actually ask, an idea WiCER proposes but didn't test. And a rule for the question the wiki can't answer: it is often a research gap rather than a wiki gap, no web search closes it, and it belongs in the research backlog as an open question.

What this means for a research wiki

Read together, the papers say less about wiki versus RAG than about how to build one, and most of it is still assumption. Ours, in one breath: compiling pays off where you synthesize and retrieval where you look things up, so hybrids are likely. The unit is the verbatim claim with its source, and the receipts are as fine-grained as the claims. Validity is a status that a recorded decision sets, not an interview. One role collects and another interprets, and answers and evidence get scored separately, by LLM judges that someone has checked against people.

The evidence is thinner than the enthusiasm. Seven of the eight ranked papers are preprints, five rest their headline numbers on scores from a language model, sometimes the same model that does the work, and none tests a wiki maintained over months on real discovery data, a real pivot or the quality of decisions. None of this makes RAG obsolete. Karpathy's team example reads to me as a design brief, not a solved problem. If you're building the memory for your own discovery work, drop me a line. We've made a fair share of the early mistakes already.