Security

Does AI Security Testing Hallucinate? LLM Guesses vs. Proof-of-Exploit

Language models hallucinate by default, and security findings are no exception. See why, and the harness, evidence rules and validation runs Agentic Deep Scan uses so a guess must become a proof before it is reported.

Mohammed Lachhab Nov 09, 2026

In May 2025, the curl project received a vulnerability report describing "a novel exploit leveraging stream dependency cycles in the HTTP/3 protocol stack", with memory corruption and possible remote code execution. It had the structure of a serious report: a protocol, a mechanism, an impact. It also referred to functions that don't exist in curl.

The report was AI-generated. curl's lead developer, Daniel Stenberg, described the problem in one sentence: "What fooled me for a short while was that it sounded almost plausible."

That is what a language model does by default. Asked to find a vulnerability, it produces the most plausible vulnerability report, and plausible is not the same as true. Every AI security testing tool runs on the same kind of model, so the question that matters is not whether the model hallucinates. It does. The question is what stands between the model's guess and your report.

This article explains why AI security testing hallucinates, why a better prompt can't fix it, and how Agentic Deep Scan is built so that a guess has to become a proof before anyone reads it as a finding.

Short answer
Yes, AI security testing hallucinates by default: every system built on a language model will sometimes produce findings that are fluent, specific and wrong. The fix is not a better model or a stricter prompt. It is a harness around the model: evidence rules every testing agent must follow, a separate re-test of every finding against the real target, and a default that downgrades anything that doesn't reproduce. The model proposes; the target's behavior decides.

What does a hallucination look like in security testing?

It rarely looks like nonsense. A hallucinated finding has the shape of a good one: a file and line number, a data flow, a severity score, a remediation snippet. That is what makes it expensive.

When we built an AI pull-request reviewer, the comments that cost the most were the ones "plausible enough that engineers felt obligated to investigate them, but incorrect enough that the investigation often wasted time". Security findings have the same problem with higher stakes, because a fake critical gets escalated.

A hallucination is also different from the false positive a rule-based scanner produces:

	Rule-based false positive	LLM hallucination	Validated finding
What triggered it	A real pattern matched in the wrong context	A gap the model filled with the most plausible story	An effect the target produced
How it reads	Terse: rule ID, line, generic description	Fluent: named components, data flow, impact, fix	Input, response or crash, control, bounded claim
What it gets wrong	The context	The facts, or the conclusion drawn from real facts	Possibly the interpretation
Cost to disprove	Usually quick: check the context	Slow: the story has to be checked claim by claim	Reviewers check the evidence instead of rebuilding it

What kinds of hallucination does AI security testing produce?

Invented code, like the functions in the curl report, is the hallucination everyone pictures. It is also the easiest kind to catch: check whether the function exists.

The harder kind is a wrong conclusion about real code. The model reads the code correctly and gets the story around it wrong: who controls the input, who can call the function, what the feature is for, and what actually happens when someone abuses it. Checking that the code exists can't catch this kind, because the code does exist.

Any AI security testing system has to defend against these patterns. The last column names the part of the Agentic Deep Scan harness that targets each one:

Hallucination	What it looks like	What in the harness targets it
Intended behavior reported as a vulnerability	An internal service that is unauthenticated by design, rated Critical	The application-contract check: establish what the product is meant to do before judging it
Code an attacker can't reach	A real bug in a function that only privileged system components can call	An explicit attacker model; severity measured as capability gained beyond existing privileges
Impact the code can't produce	A heap overflow whose write is in fact bounds-checked	Medium and above requires demonstrated impact; a separate validation run has to reproduce it
Test code treated as shipped code	A flaw in a unit-test helper that isn't in the release build	Static findings need a complete path from attacker-controlled input to the sink
Invented components or citations	A content provider, endpoint or function that doesn't exist	Evidence grounding: nothing exists unless a tool output shows it
Test-environment artifacts	A key reported as software-backed because the test ran on an emulator without hardware key storage	Status recorded per task; a proof only holds where it ran
Proof only in the prose, or staged	"The PoC succeeded" pointing to a file that was never saved; a PoC that plants the attacker's foothold by hand	Cited files must be opened and read; simulated results can't confirm; the validation run re-tests the claim

Why do language models hallucinate vulnerabilities?

Researchers at OpenAI and Georgia Tech argue that "language models hallucinate because the training and evaluation procedures reward guessing over acknowledging uncertainty". In security work, a guess has a familiar shape: input reaches a sink, so there is code execution. That sentence is the most plausible continuation of a lot of code, whether or not it is true.

Five things make it worse in security testing:

The threat model isn't in the code. Source code shows that a value reaches a sink. It doesn't show whether an attacker controls that value, whether the caller is already trusted, or whether the behavior is the product's purpose. Those facts live in the design and the deployment.
Gaps get filled with the typical case. When the model can't see a permission declaration, a dependency's source or the runtime configuration, it assumes the common one.
Rule hits act as anchors. A secret-scanner hit or an exported component is a fact the model is asked to explain. It will explain it, even when the explanation needs components that don't exist.
The task rewards findings. An agent asked to find vulnerabilities is under pressure to produce them, and long sessions weaken the instructions meant to restrain it. We described the same effect in our post-mortem on agent scope drift.
The reasoning isn't stable. An evaluation of eight LLMs on 228 code scenarios, published at IEEE S&P 2024, found "non-deterministic responses, incorrect and unfaithful reasoning". Renaming functions or variables produced incorrect answers in 26% of cases. Adding library functions did so in 17%.

Open-source maintainers see the result from the receiving end. When curl ended its bug bounty on January 31, 2026, its maintainers cited a flood of AI-generated reports among the reasons. The share of reports that turned out to be real vulnerabilities had fallen from above 15% in earlier years to below 5% from 2025. In the announcement's words: "The never-ending slop submissions take a serious mental toll to manage."

Why can't a prompt, a confidence score or a second model fix it alone?

Each of these consults a model again, so each inherits the model's blind spots.

"Avoid false positives" prompts. A prompt that asks for fewer false positives doesn't remove them. Defining the minimum evidence for each vulnerability class does.
Confidence scores. A confidence score is the model's statement about itself.
Asking again. Sampling the same model several times and taking the majority helps with random errors. It doesn't help when every sample shares the same missing context.
A second model as judge. A judge that reads the first model's explanation inherits its story, so it is not independent evidence.

Rules for the model are still necessary, and Agentic Deep Scan has strict ones. But a rule in a prompt is followed most of the time, not every time. So in Agentic Deep Scan, every rule is backed by a check that does not depend on the model obeying it: a re-test against the real target, a default that downgrades what doesn't reproduce, and limits enforced by the system.

Anything that asks a model again checks the model against itself. Proof checks the model against the target.

What is proof-of-exploit, and why can't it be hallucinated?

What is proof-of-exploit?
Proof-of-exploit is evidence that a vulnerability actually fired: a replayable input sent to the real target that produces an observable effect, which only happens if the flaw exists. The test records the effect, and a control shows that it disappears without the exploit.

The model can still choose what to test. It doesn't get to decide what happened. That decision belongs to an oracle, something you can observe that separates "the vulnerability fired" from "something happened":

Vulnerability class	Oracle	Example
Memory corruption	Sanitizer crash on the vulnerable build, clean exit on the patched build	CyberGym credits a task only when the PoC crashes the unpatched build and exits cleanly on the fixed one
Blind code execution	A delay that tracks the payload	A 10-second sleep in a Langflow component returned in 40.50 seconds; the harmless version returned in 0.43
Server-side request forgery	An out-of-band callback from the application server, not the tester's browser	A request logged by a server the tester controls
Broken access control	Another account's data in the response, while the direct request is refused	A banking chatbot revealed another user's balance while the direct API returned `403 Forbidden`
Unauthorized action	A state change confirmed by an independent read	The same chatbot changed a payment destination for a read-only user; a separate read confirmed the change

Checking that the cited code exists catches invented code and nothing else. It can't tell whether an attacker controls the input, whether they can reach the code, or what happens when they do. An exploit run from an attacker's position answers all three: the effect either happens with attacker-level access, or it doesn't.

	LLM guess	Proof-of-exploit
Source of truth	The model's reading of code or responses	The target's behavior
Who produces the evidence	The model, in prose	The target, through tools that actually ran
Can someone else replay it?	No: they have to investigate again	Yes: same input, same effect
What it can claim	That something might be vulnerable	That this input produced this effect, under these conditions
How it fails	Invented facts, or wrong conclusions about real code	A real effect, misread
What the developer does next	Investigates	Fixes, then re-runs the same proof as a regression test

Our full standard for calling a finding validated has five parts: a reproducible artifact, an oracle, a negative control, preconditions tested by removing them, and an honest boundary between what was shown and what was inferred.

How does the Agentic Deep Scan harness keep hallucinations out of the report?

Agentic Deep Scan is Ostorlab's AI pentesting scan for web applications, APIs, mobile apps and source code: AI agents plan and run tests against the target, and a separate validation run re-tests every finding before it is reported as confirmed.

We built Agentic Deep Scan on the assumption that the model will hallucinate, so no single step is trusted to catch it. The harness, not the model, decides what counts. It stacks six layers between a model's guess and a reported finding, and the later layers re-test what the earlier ones concluded:

Six layers stand between a model's hypothesis and a verified finding: scope and guardrails, the planner's attacker model, the executor's evidence rules, the rulebook, a separate validation run, and people — Figure 1: The six layers of the Agentic Deep Scan harness between a model's hypothesis and a verified finding

Layer 1: scope and guardrails, before the scan starts

Every agentic scan has a Guardrails step that sets the approved scope and the environments to exclude. It also sets how the agent handles risky actions: you can require it to ask first when an action reaches real people, is destructive or leaves the scope, and when it cannot tell whether a target is in scope. If a scope question gets no answer, the agent must treat the target as out of scope.

That matters for hallucination because a model that guesses "this is probably in scope" is hallucinating with consequences. The hard limits don't depend on the model's judgment at all: the system enforces the IP addresses scan traffic must never reach, and the request rate.

Layer 2: the planner works from an explicit attacker model

Planners decide what to test next. Where the attacker's position decides the verdict, the planner spells out who the attacker is. For abuse of an Android app's content provider or Binder interface, the default attacker is "a separately installed, non-root Android app with its own UID and no shared signing key". Root, run-as and debuggable-only access cannot be used to prove the abuse, because the test device grants those to the tester and never to an attacker.

Our source-code planners for injection, SSRF, template injection and native-code bugs close each candidate with one of five dispositions: confirmed, refuted, incomplete trace, runtime-dependent, or not applicable. "We could not finish the trace" is a recorded outcome, not a gap the model gets to fill with a story.

Layer 3: the executor follows evidence rules it can't skip

The executors that run tools against the target get a set of evidence rules added to their instructions in code, so no specialized executor can leave them out:

Never fabricate tool output. Every finding has to be backed by output from a tool that actually ran.
A class name or a string does not prove that an endpoint, route or API exists. Only exact paths in tool output count.
Quote the exact output that supports a finding. If it can't be quoted, the finding isn't confirmed.
Label anything without concrete proof as theoretical or unconfirmed.
An empty search proves only that the search found nothing in what it looked at.
Test the supplied target. Mock services, invented data, generated success messages and simulated responses are not evidence about the target, even when the script that produced them ran successfully.

Every task ends with a status: success, simulated, failed, blocked or unsupported. Any status other than success comes with the concrete reason. The task also records reproduction steps from a clean state, using the exact requests or commands that ran, and its tool calls and outputs are recorded as the scan runs.

Layer 4: a rulebook for confirming, and for refuting

After testing, a separate analysis step reads the full execution history, not the executor's summary of it, and decides what becomes a finding. Its rules target the common hallucination patterns:

Only successful tasks count. Simulated results support "Potentially" at most. Failed, blocked and unsupported tasks cannot be cited as evidence.
Claims without raw output are dropped. If the history says an endpoint, file or behavior exists but no tool output shows it, the finding is unconfirmed. Files mentioned in the history must be opened and read before they are cited, because a filename is not evidence.
Establish the application's contract first. Before judging a finding, determine what the product is meant to do, who is allowed to do it, and what it guarantees, researching vendor documentation when unsure. Behavior inside the documented contract is intended, not a vulnerability.
Severity follows demonstrated impact. Severity is the capability an attacker gains beyond the privileges they already have. Medium or above requires demonstrated impact; a condition without it is capped at Low, Hardening or Info, and the report says that no impact was shown.
No hypothetical evidence. Phrases like "would result in" or "could be retrieved" don't belong in exploitation evidence. Evidence from static analysis only, or inferred rather than observed, is labeled as such.
An empty list is a valid answer. A blocked or failed test certifies nothing: it becomes neither a finding nor a clean bill of health.

The rulebook works in the other direction too. A model in a hurry can hallucinate safety as easily as a vulnerability, and a wrongly closed finding is a miss nobody sees. So our native-code planners are not allowed to close a candidate as safe because:

an input exited cleanly without ever reaching the vulnerable code;
a static analyzer reported nothing, which is missing coverage, not proof of safety;
a guard exists, but it bounds a different value than the one that causes the defect;
only the default variant of a format or protocol was tested.

A proof-of-concept that didn't trigger never turns a candidate into Secure either: the finding is rated on its static evidence alone and marked as dynamically unconfirmed. In the words of the rule: "Difficulty reaching a defect is not a refutation of it."

Layer 5: a separate validation run re-tests every finding

Every finding rated from Hardening to Critical goes to a validation run. The validation run gets the finding as written, not the history and reasoning that produced it. It builds its own plan, runs on a separately configured validation model, and tests the real target for up to 40 iterations. It is recorded as its own risk in the scan, with its own plan and tool calls, and its findings are extracted only from what it actually executed.

If the finding reproduces, it is reported with the validation evidence attached. If it doesn't, it is downgraded to Potentially, together with a specific reason that cites what the validation observed, for example that every test request returned 403 with no database error. If the validation run fails with an error, the finding is downgraded too. A finding is never promoted because validation could not run.

Layer 6: people, re-tests and evaluation

The last layer is outside the model entirely:

People at the gates. With approvals enabled, risky actions wait for a person. Every finding carries its exploitation evidence and reproduction steps, so a person can challenge the evidence before acting on it instead of reconstructing the investigation.
Re-tests on demand. Dig Deeper and Single Vulnerability Assessment re-investigate one finding, and Risk Reruns repeat an investigation with a different model or more effort.
Continuous evaluation. We measure the agent against targets where the answer is known, from benchmarks such as CyberGym to open-source projects whose source we check findings against. Every mistake we find becomes a new rule in the harness.

The same principle runs through the rest of the product: in Precheck, the model judges each screen, and the code decides whether the scan failed.

Can proof-of-exploit be wrong too?

Yes, and a vendor that claims "zero hallucinations" is hiding this. Proof-of-exploit can fail in four ways, and the harness is built partly to catch them:

A proof described in a sentence, or staged. "The PoC succeeded" is a claim, and a PoC that sets up the attacker's position by hand proves only the setup. Proof is output from a tool that ran against the target: a request and its response, an input file and its crash log, a callback and the server log.
The right effect in the wrong environment. A key that looks software-backed on an emulator says nothing about real devices with hardware key storage. A proof only holds where it ran, so the report has to name the build, device and version.
The right effect, but no boundary crossed. An exploit that works for an already-privileged caller, or with the test device's root access, demonstrates a bug. It doesn't demonstrate a vulnerability.
No control. A slow response can come from load, and a 200 instead of a 403 can come from a cache. In one mobile authorization case, we repeated the test with a fresh credential to rule out a cached response.

The reverse also holds: a failure to reproduce in a constrained test does not prove the candidate was a false positive. That is why an unreproduced finding becomes Potentially rather than disappearing.

Proof-of-exploit removes invented claims. It does not remove misread ones. That is why the layers stack, and why a person should review the evidence before acting on a material finding.

How can you tell whether an AI security finding is a guess or a proof?

Whichever tool produced the finding, these eight questions separate the two:

Do all the components, endpoints and file:line references it names exist in the target as tested?
Is there a replayable artifact (a request, an input file, a PoC app), or only a description?
What is the oracle, and was it observed in tool output or only described by the model?
Is there a negative control showing the effect disappears without the exploit?
Which trust boundary was crossed, and from what privilege level?
Does the test environment match what ships: build, version, device?
Does the claim stop where the evidence stops?
Can the vendor show you the hypotheses it rejected or downgraded, and why?

The last question is the most revealing. A system that validates findings produces a lot of rejected and downgraded hypotheses. A vendor that can't show any probably isn't validating.

Treat these as red flags:

Claims of "zero false positives" or "zero hallucinations".
A confidence percentage presented as evidence.
A critical severity with no reproduction.
Citations to code you can't find.

So, does AI security testing hallucinate?

Yes, by default and permanently. Every system built on a language model will produce findings that are fluent, specific and wrong, and no prompt or model upgrade removes that.

What separates AI security testing tools is what they do about it. Agentic Deep Scan treats every model output as a hypothesis: evidence rules the testing agents must follow, an explicit attacker model, a separate re-test against the real target, a default that downgrades what doesn't reproduce, and limits the model cannot talk its way past. A guess only becomes a finding when the target confirms it.

The model can guess. The exploit has to work.

To see the harness work on your own application, run Agentic Deep Scan on your web applications and APIs or your Android and iOS apps. Every confirmed finding arrives with the exploitation evidence and reproduction steps behind it.

FAQ

Do AI pentesting tools hallucinate vulnerabilities?

Yes. Every tool built on a language model can produce findings that are fluent, specific and wrong. Sometimes the model invents code that isn't there. More often it reads real code correctly and draws the wrong conclusion: intended behavior reported as a flaw, code an attacker can't reach, or an impact the code can't produce. The difference between tools is what stops such a finding from reaching the report.

How does Ostorlab keep AI hallucinations out of its findings?

Agentic Deep Scan assumes the model will be wrong and stacks several checks in its harness. Scope and approvals are set before the scan. Every executor must back findings with verbatim output from tools that actually ran. A separate analysis step applies an evidence rulebook, including a check of the application's intended behavior. Every finding rated Hardening or above is then re-tested in a separate validation run against the real target. Findings that don't reproduce are downgraded to Potentially with the reason, and every reported finding carries the evidence a person needs to check it.

What is the difference between an AI hallucination and a false positive?

A rule-based false positive is a real pattern matched in the wrong context, such as a dangerous function called with a constant. A hallucination is a fabricated fact or conclusion presented as a confident narrative: a component that doesn't exist, an attacker who can't actually reach the code, or an impact the code can't produce. Hallucinations cost more to disprove because every claim in the story has to be checked.

What is proof-of-exploit?

Proof-of-exploit is evidence that a vulnerability fired: a replayable input sent to the real target that produces an observable effect, such as a sanitizer crash, an out-of-band callback or another account's data, which only happens if the flaw exists. The effect is recorded in the output of the tool that ran, and a negative control shows it disappears without the exploit.

Can an AI agent fake a proof of concept?

It can claim one. A model can write that a PoC succeeded and point to files that were never saved, or stage the attacker's position by hand. That is why proof has to be tool output from a test against the real target that someone else can replay, why cited files have to be opened before they count, and why a separate validation run re-tests the claim instead of re-reading it.

Can a second AI model verify an AI finding?

It can help with triage, but it is not independent evidence if it reads the first model's explanation. A separate run that re-tests the finding against the target is a different thing: it checks the claim against the target's behavior rather than against another model's opinion.

Does proof-of-exploit eliminate false positives?

It eliminates invented findings, but not misread ones. A real effect can be misinterpreted if the test ran in the wrong environment, used privileges an attacker doesn't have, or lacked a negative control. Human review of the evidence is still needed before acting on a material finding.

Which findings can't be proven by exploitation?

Some issues are real but have no single observable effect: weak cryptographic choices, missing hardening, or business-logic risks whose impact depends on context. These should be reported as Potentially, Low, Hardening or Info, with what was shown and what was inferred stated separately, rather than rounded up to a confirmed exploit.

References

Tags:

AI Pentesting, Hallucination, Proof-of-Exploit, Agentic AI, AppSec, False Positives, Agentic Deep Scan