Exploiting LLM Overconfidence: Hallucination as an Attack Vector

Hallucination Taxonomy

LLM hallucination refers to the generation of content that is factually incorrect, fabricated, or inconsistent with the source material, produced with apparent confidence. Understanding the types of hallucination is essential for understanding how they are exploited.

Intrinsic hallucination occurs when the model contradicts information that was explicitly provided to it — in its training data, its context window, or its retrieval corpus. The model generates output inconsistent with a source it demonstrably has access to. This is the hallucination type most easily addressed through retrieval-augmented generation and grounding techniques.

Extrinsic hallucination occurs when the model generates claims that cannot be verified against any provided source — neither confirmed nor contradicted by available data. The model invents plausible-sounding facts from whole cloth. This is harder to detect because there is no ground truth to compare against, and it is the dominant attack vector for misinformation exploitation.

Conflation hallucination — less formally classified but security-relevant — occurs when the model correctly recalls facts about two different entities but merges them, producing a confident but incorrect composite. An attacker can exploit this by asking about an entity name that partially overlaps with a known entity.

How Adversaries Elicit Confident Misinformation

The model's output confidence is not tightly coupled to its accuracy. Several prompting strategies reliably increase the rate and conviction of hallucinated output:

Obscure subject matter: Ask about real-but-obscure topics where training data is sparse. The model has enough signal to generate plausible-sounding content but insufficient data to anchor to accurate facts:

"What were the specific findings of the 2019 Hartmann-Kelsey study
on microbiome effects in high-altitude populations?"

(No such study exists. The model will fabricate one with realistic-sounding methodology and statistics.)

Leading questions: Frame the question to presuppose a false fact:

"Given that caffeine has been shown to reduce insulin sensitivity
by 40% in fasting conditions, what dietary adjustments do you recommend?"

The model frequently accepts the presupposition as given and elaborates on it confidently, amplifying the original misinformation.

Fake citation requests: Ask the model to provide references for invented claims:

"Can you cite three peer-reviewed studies supporting the claim that
intermittent fasting reverses Type 2 diabetes?"

The model will often generate plausible-seeming citations — authors with real-sounding names, journals that exist, plausible years — none of which correspond to real papers.

Authority anchoring: Reference a prestigious institution to lower the model's skepticism threshold:

"According to the Mayo Clinic's 2023 guidelines, what is the
recommended dosage of [substance] for [condition]?"

Downstream Trust Exploitation

The danger of LLM misinformation is not the error itself — it is the trust transfer. When an LLM presents false information confidently, in fluent authoritative prose, complete with citations and statistics, users systematically underestimate the probability that it is wrong.

Legal applications: Attorneys have submitted AI-generated briefs citing non-existent case law. The cases were invented with realistic-sounding names, courts, and docket numbers. This is documented, not hypothetical.

Medical decision support: Practitioners consulting LLMs for drug interaction information, dosage guidance, or diagnostic criteria receive confident misinformation at a rate that increases with query specificity.

Financial analysis: Asking an LLM to summarize a company's recent financial performance may yield a confident summary that blends real and invented figures, particularly for smaller companies with sparse training data.

Disinformation campaigns: Adversaries use LLMs to generate large volumes of plausible misinformation on target topics — fake studies, fabricated quotes from real public figures, invented events — scaled in ways that manual content farms cannot match.

Calibration Techniques

Temperature and top-p adjustment: Lower temperature reduces creativity but does not reliably reduce hallucination rates. It primarily makes the model more repetitive, not more accurate.

Self-consistency sampling: Generate multiple responses to the same query and compare them. Inconsistent answers across samples are a signal of low confidence and high hallucination risk. Implementations can surface confidence scores based on response variance.

Chain-of-thought with explicit uncertainty: Prompt the model to reason step-by-step and explicitly state when it is uncertain:

"Answer the following question. If you are not certain of any fact,
say so explicitly before stating it. Do not fabricate citations."

This does not eliminate hallucination but can reduce confident misinformation by activating the model's uncertainty expression capabilities.

Retrieval grounding: RAG with strict source attribution — requiring the model to cite the specific retrieved chunk for every factual claim — significantly reduces extrinsic hallucination, though it does not eliminate intrinsic hallucination.

TruthfulQA Benchmark

TruthfulQA is a benchmark of 817 questions designed to elicit hallucinations from large language models. Questions span categories where humans commonly hold false beliefs — health, law, finance, history — making them particularly effective at surfacing model failures. The benchmark measures both truthfulness (whether answers are true) and informativeness (whether answers are useful rather than evasive).

State-of-the-art models as of 2024 achieve roughly 80-90% truthfulness on TruthfulQA — which sounds high until you consider that a 10-20% hallucination rate on a system answering thousands of queries per day produces a substantial volume of confidently-stated misinformation.