
On this page
Ask how often AI tools invent citations and you get a number with no context: 55%, or 18%, or "it's much better now." All three are true at once — different models, different tasks, different definitions of fabricated, different years.
Seven studies are in the table below, each with the sample behind its rate and its own definition of a miss. The rates run from 6% to 91.4%, and that spread is not a finding: the seven studies did not count the same thing.
Last reviewed October 25, 2026: three rows added, five sample sizes corrected, two rates put back inside the limits their studies set, and the claim to hold every study withdrawn.
The Table
| Study | Model | Miss rate | Miss type | Task |
|---|---|---|---|---|
| Walters & Wilder, 2023 | GPT-3.5 | 55% of 222 | No such work | Short reviews, 42 multidisciplinary topics |
| Walters & Wilder, 2023 | GPT-4 | 18% of 414 | No such work | Short reviews, 42 multidisciplinary topics |
| Chelli et al., 2024 | GPT-3.5 | 39.6% of 139 | Ground truth | 11 published reference lists, rotator cuff pathology |
| Chelli et al., 2024 | GPT-4 | 28.6% of 119 | Ground truth | 11 published reference lists, rotator cuff pathology |
| Chelli et al., 2024 | Bard | 91.4% of 104 | Ground truth | 11 published reference lists, rotator cuff pathology |
| Mugaanyi et al., 2024 | GPT-3.5 | 61.8% of 55 | DOI invalid | Introductions, 5 natural-science topics |
| Mugaanyi et al., 2024 | GPT-3.5 | 89.4% of 47 | DOI invalid | Introductions, 5 humanities topics |
| Linardon et al., 2025 | GPT-4o | 6% of 68 | No source | Reviews, major depressive disorder |
| Linardon et al., 2025 | GPT-4o | 28% of 60 | No source | Reviews, binge eating disorder |
| Linardon et al., 2025 | GPT-4o | 29% of 48 | No source | Reviews, body dysmorphic disorder |
| Kim, Kipchumba & Min, 2025 ‡ | 4 models: GPT-4o-mini, Claude 3 Haiku, Gemini 2.0 Flash-Lite, DeepSeek V3 | above 80% of an unstated lower-income share † | DOI vs Crossref | Standard prompts, 10 countries across income levels |
| Cabezas-Clavijo & Sidorenko-Bautista, 2026 ‡ | 8 chatbots, free versions | 39.8% of 400 | 5 components | One prompt, 5 knowledge areas |
| Ülkir & Paslı, 2026 ‡ | ChatGPT 5.2 | 23.2% of an unstated share † | Components + PMID | 120 open-ended anatomy questions |
| Ülkir & Paslı, 2026 ‡ | Gemini 3 Pro | 45.8% of an unstated share † | Components + PMID | 120 open-ended anatomy questions |
| Ülkir & Paslı, 2026 ‡ | DeepSeek V3.2 | 47.5% of an unstated share † | Components + PMID | 120 open-ended anatomy questions |
† The denominator behind this rate is not published. Kim, Kipchumba, and Min's figure covers only the lower-income countries in a set of 3,451 validated citations; Ülkir and Paslı report 1,800 references across three models with no per-model count.
‡ Read from the study's abstract only, on October 25, 2026.
The labels in the Miss type column are ours. The criterion beside each one is the study's:
| Miss type | A reference counts as a miss when | Study |
|---|---|---|
| No such work | No real paper matches both the title and the authors | Walters & Wilder |
| Ground truth | Any two of title, first author, and year are wrong against the published list the model is asked to rebuild | Chelli et al. |
| DOI invalid | The DOI is absent, invalid, or matches a different source | Mugaanyi et al. |
| No source | No identifiable source stands behind the reference | Linardon et al. |
| DOI vs Crossref | The DOI fails validation against the Crossref API | Kim, Kipchumba & Min |
| 5 components | Any one of authorship, year, title, source, or location is wrong or invented | Cabezas-Clavijo & Sidorenko-Bautista |
| Components + PMID | Components including the PMID fail against indexed sources | Ülkir & Paslı |
What These Numbers Do Not Compare
Which Model You Open Matters More Than Which Year It Is
Walters and Wilder ran the same 42 topics through two models one generation apart, and fabrication fell from 55% to 18% with everything else held constant.
Chelli and colleagues ran three models against 11 published reference lists a year later: GPT-3.5 missed on 39.6% and GPT-4 on 28.6% — a fall in the same direction Walters and Wilder measured, though smaller — and Bard on 91.4%. Subtract: 62.8 points between two models in one test, against 37 points between GPT-3.5 and GPT-4 a generation apart. The widest gap any single test in this table records is not between a 2023 model and a 2026 one; it sits between two models tested together in 2024.
Ülkir and Paslı measured a spread between contemporaries again two years later: 120 anatomy questions, three models, 23.2%, 45.8%, and 47.5%.
Cabezas-Clavijo and Sidorenko-Bautista ran one prompt past eight free chatbots: of 400 references, 26.5% were fully correct, 33.8% partially correct, and 39.8% erroneous or fabricated outright.
Fabrication Is Not the Only Error Mode
Walters and Wilder also checked the citations that were not fabricated — the ones pointing at genuine papers — and found substantive errors in 43% of GPT-3.5's real citations and 24% of GPT-4's: wrong year, wrong journal, wrong page range, wrong author order. Put the two together: 82% of GPT-4's citations were real, and three-quarters of those were clean, so about 62% were both.
A different study found the same error mode under its own definition. Of the 176 citations GPT-4o produced for Linardon and colleagues, 141 pointed at real papers, and 64 of those — 45.4% — carried an error, most often an incorrect or invalid DOI. A reference list that passes a fabrication check is not a reference list that passes.
"Hallucination" Is Not Defined the Same Way Twice
Some studies count a citation as hallucinated when the paper does not exist; others when the DOI does not resolve, which is stricter: a real paper carrying an invented DOI passes the first test and fails the second. The Mugaanyi and Kim rows measure DOIs, not papers — a different measurement, not a worse model.
A third definition measures against a known ground truth. Chelli and colleagues asked each model to rebuild a published systematic review's reference list, counting a reference as hallucinated when any two of title, first author, and year were wrong. That is closer to what a literature review actually asks of a tool.
DeepSeek shows what this does to any ranking. Cabezas-Clavijo and Sidorenko-Bautista name V3 as one of two chatbots in their test that produced no false references; Ülkir and Paslı measured V3.2 missing on 47.5% of its anatomy references. Same family, different version, different task, different definition of a miss. Read the table as seven studies that measured seven different things, not fifteen numbers on one scale.
What Predicts a High Rate
Three variables come out of these studies. Each rests on a single study — two of them on the same one — and all three are knowable before you prompt.
Topic obscurity. Linardon and colleagues held the model constant and compared three disorders of different research maturity: 6% for major depressive disorder, 28% for binge eating disorder, and 29% for body dysmorphic disorder. The authors offer a reason for that order and mark it as a hypothesis, not a finding: topics with limited research coverage are underrepresented in the training data, so there is less to retrieve.
Prompt specificity. In the same study, asking for specialized material on binge eating disorder pushed fabrication to 46%, against 17% for a general overview of the same disorder. The effect did not generalize: across the study as a whole, general and specialized reviews did not differ significantly, and for body dysmorphic disorder the specialized reviews were the more accurate ones. A narrow request on a thinly researched topic is where the two risks meet.
Geography. Kim, Kipchumba, and Min validated 3,451 citations from four models against the Crossref API across 10 countries and found DOI hallucination above 80% for the lower-income ones, with structurally complete but invalid citations especially frequent for countries such as India and Bangladesh. The same study ranks its predictors: the model matters most, then income level, then how recent the work is.
Line the three up: how much has been published on your topic, how narrow your question is, and which country the work comes from.
How We Maintain This Page
We add a study when it is peer-reviewed or a preprint from an identifiable research group, reports a sample size, and states what it counted as a miss. A study reporting a composite score that folds citation accuracy into other measures, or a rate per answer rather than per reference, does not get a row. We record the model, task, sample, and rate as the authors reported them, and do not rescale numbers to make them comparable, because they are not. Where a rate's denominator is not published, the rate carries †; where only the abstract was readable, the study carries ‡.
If you find an error here, or a study we have missed, write to [email protected] and we will fix the page, with the date of the change. Those five sample sizes had been taken from a study's total rather than the denominator behind each rate. Writing about citation accuracy obliges us to be correctable in public.
The Rate Will Not Tell You Which Citation Is Invented
Finding which of your own citations is the invented one needs a per-reference check: whether the DOI resolves, whether the record matches field by field (a real DOI can point at another paper), and whether the listed authors appear on that work (a fabricated title can sit under a real name). That is what it means to check AI citations rather than count them.
The fix is in the order of the work: save the record first, open it, then write the sentence it supports. That order puts the checking before the draft instead of after it.
Sources
The seven studies were published between 2023 and 2026 in Scientific Reports, the Journal of Medical Internet Research (two of them), JMIR Mental Health, Publications, the Journal of Data and Information Science, and Clinical Anatomy. Each row links to the study it reports.