citationsAI researchacademic integrity

AI Citation Error Rates: Seven Studies in One Table

AI citation error rates run 6% to 91.4%, and that spread is not one scale: seven studies counted seven different kinds of miss, each with its sample.

An error rate alone tells you little: the model matters more than the year, fabrication is not the only error, and definitions differ by study.
On this page

Ask how often AI tools invent citations and you get a number with no context: 55%, or 18%, or "it's much better now." All three are true at once — different models, different tasks, different definitions of fabricated, different years.

Seven studies are in the table below, each with the sample behind its rate and its own definition of a miss. The rates run from 6% to 91.4%, and that spread is not a finding: the seven studies did not count the same thing.

Last reviewed October 25, 2026: three rows added, five sample sizes corrected, two rates put back inside the limits their studies set, and the claim to hold every study withdrawn.

The Table

StudyModelMiss rateMiss typeTask
Walters & Wilder, 2023GPT-3.555% of 222No such workShort reviews, 42 multidisciplinary topics
Walters & Wilder, 2023GPT-418% of 414No such workShort reviews, 42 multidisciplinary topics
Chelli et al., 2024GPT-3.539.6% of 139Ground truth11 published reference lists, rotator cuff pathology
Chelli et al., 2024GPT-428.6% of 119Ground truth11 published reference lists, rotator cuff pathology
Chelli et al., 2024Bard91.4% of 104Ground truth11 published reference lists, rotator cuff pathology
Mugaanyi et al., 2024GPT-3.561.8% of 55DOI invalidIntroductions, 5 natural-science topics
Mugaanyi et al., 2024GPT-3.589.4% of 47DOI invalidIntroductions, 5 humanities topics
Linardon et al., 2025GPT-4o6% of 68No sourceReviews, major depressive disorder
Linardon et al., 2025GPT-4o28% of 60No sourceReviews, binge eating disorder
Linardon et al., 2025GPT-4o29% of 48No sourceReviews, body dysmorphic disorder
Kim, Kipchumba & Min, 2025 ‡4 models: GPT-4o-mini, Claude 3 Haiku, Gemini 2.0 Flash-Lite, DeepSeek V3above 80% of an unstated lower-income share †DOI vs CrossrefStandard prompts, 10 countries across income levels
Cabezas-Clavijo & Sidorenko-Bautista, 2026 ‡8 chatbots, free versions39.8% of 4005 componentsOne prompt, 5 knowledge areas
Ülkir & Paslı, 2026 ‡ChatGPT 5.223.2% of an unstated share †Components + PMID120 open-ended anatomy questions
Ülkir & Paslı, 2026 ‡Gemini 3 Pro45.8% of an unstated share †Components + PMID120 open-ended anatomy questions
Ülkir & Paslı, 2026 ‡DeepSeek V3.247.5% of an unstated share †Components + PMID120 open-ended anatomy questions

† The denominator behind this rate is not published. Kim, Kipchumba, and Min's figure covers only the lower-income countries in a set of 3,451 validated citations; Ülkir and Paslı report 1,800 references across three models with no per-model count.

‡ Read from the study's abstract only, on October 25, 2026.

The labels in the Miss type column are ours. The criterion beside each one is the study's:

Miss typeA reference counts as a miss whenStudy
No such workNo real paper matches both the title and the authorsWalters & Wilder
Ground truthAny two of title, first author, and year are wrong against the published list the model is asked to rebuildChelli et al.
DOI invalidThe DOI is absent, invalid, or matches a different sourceMugaanyi et al.
No sourceNo identifiable source stands behind the referenceLinardon et al.
DOI vs CrossrefThe DOI fails validation against the Crossref APIKim, Kipchumba & Min
5 componentsAny one of authorship, year, title, source, or location is wrong or inventedCabezas-Clavijo & Sidorenko-Bautista
Components + PMIDComponents including the PMID fail against indexed sourcesÜlkir & Paslı

What These Numbers Do Not Compare

Which Model You Open Matters More Than Which Year It Is

Walters and Wilder ran the same 42 topics through two models one generation apart, and fabrication fell from 55% to 18% with everything else held constant.

Chelli and colleagues ran three models against 11 published reference lists a year later: GPT-3.5 missed on 39.6% and GPT-4 on 28.6% — a fall in the same direction Walters and Wilder measured, though smaller — and Bard on 91.4%. Subtract: 62.8 points between two models in one test, against 37 points between GPT-3.5 and GPT-4 a generation apart. The widest gap any single test in this table records is not between a 2023 model and a 2026 one; it sits between two models tested together in 2024.

Ülkir and Paslı measured a spread between contemporaries again two years later: 120 anatomy questions, three models, 23.2%, 45.8%, and 47.5%.

Cabezas-Clavijo and Sidorenko-Bautista ran one prompt past eight free chatbots: of 400 references, 26.5% were fully correct, 33.8% partially correct, and 39.8% erroneous or fabricated outright.

Fabrication Is Not the Only Error Mode

Walters and Wilder also checked the citations that were not fabricated — the ones pointing at genuine papers — and found substantive errors in 43% of GPT-3.5's real citations and 24% of GPT-4's: wrong year, wrong journal, wrong page range, wrong author order. Put the two together: 82% of GPT-4's citations were real, and three-quarters of those were clean, so about 62% were both.

A different study found the same error mode under its own definition. Of the 176 citations GPT-4o produced for Linardon and colleagues, 141 pointed at real papers, and 64 of those — 45.4% — carried an error, most often an incorrect or invalid DOI. A reference list that passes a fabrication check is not a reference list that passes.

"Hallucination" Is Not Defined the Same Way Twice

Some studies count a citation as hallucinated when the paper does not exist; others when the DOI does not resolve, which is stricter: a real paper carrying an invented DOI passes the first test and fails the second. The Mugaanyi and Kim rows measure DOIs, not papers — a different measurement, not a worse model.

A third definition measures against a known ground truth. Chelli and colleagues asked each model to rebuild a published systematic review's reference list, counting a reference as hallucinated when any two of title, first author, and year were wrong. That is closer to what a literature review actually asks of a tool.

DeepSeek shows what this does to any ranking. Cabezas-Clavijo and Sidorenko-Bautista name V3 as one of two chatbots in their test that produced no false references; Ülkir and Paslı measured V3.2 missing on 47.5% of its anatomy references. Same family, different version, different task, different definition of a miss. Read the table as seven studies that measured seven different things, not fifteen numbers on one scale.

What Predicts a High Rate

Three variables come out of these studies. Each rests on a single study — two of them on the same one — and all three are knowable before you prompt.

Topic obscurity. Linardon and colleagues held the model constant and compared three disorders of different research maturity: 6% for major depressive disorder, 28% for binge eating disorder, and 29% for body dysmorphic disorder. The authors offer a reason for that order and mark it as a hypothesis, not a finding: topics with limited research coverage are underrepresented in the training data, so there is less to retrieve.

Prompt specificity. In the same study, asking for specialized material on binge eating disorder pushed fabrication to 46%, against 17% for a general overview of the same disorder. The effect did not generalize: across the study as a whole, general and specialized reviews did not differ significantly, and for body dysmorphic disorder the specialized reviews were the more accurate ones. A narrow request on a thinly researched topic is where the two risks meet.

Geography. Kim, Kipchumba, and Min validated 3,451 citations from four models against the Crossref API across 10 countries and found DOI hallucination above 80% for the lower-income ones, with structurally complete but invalid citations especially frequent for countries such as India and Bangladesh. The same study ranks its predictors: the model matters most, then income level, then how recent the work is.

Line the three up: how much has been published on your topic, how narrow your question is, and which country the work comes from.

How We Maintain This Page

We add a study when it is peer-reviewed or a preprint from an identifiable research group, reports a sample size, and states what it counted as a miss. A study reporting a composite score that folds citation accuracy into other measures, or a rate per answer rather than per reference, does not get a row. We record the model, task, sample, and rate as the authors reported them, and do not rescale numbers to make them comparable, because they are not. Where a rate's denominator is not published, the rate carries †; where only the abstract was readable, the study carries ‡.

If you find an error here, or a study we have missed, write to [email protected] and we will fix the page, with the date of the change. Those five sample sizes had been taken from a study's total rather than the denominator behind each rate. Writing about citation accuracy obliges us to be correctable in public.

The Rate Will Not Tell You Which Citation Is Invented

Finding which of your own citations is the invented one needs a per-reference check: whether the DOI resolves, whether the record matches field by field (a real DOI can point at another paper), and whether the listed authors appear on that work (a fabricated title can sit under a real name). That is what it means to check AI citations rather than count them.

The fix is in the order of the work: save the record first, open it, then write the sentence it supports. That order puts the checking before the draft instead of after it.

Sources

The seven studies were published between 2023 and 2026 in Scientific Reports, the Journal of Medical Internet Research (two of them), JMIR Mental Health, Publications, the Journal of Data and Information Science, and Clinical Anatomy. Each row links to the study it reports.