If an AI agent produces a convincing exploit narrative, the finding can be trusted.

No. A plausible explanation is still only a lead.

During an authorized mobile application scan, our workflow surfaced what looked like an authorization failure in an account flow. The client appeared to use an application-level credential for backend calls and a second, account-specific credential for sensitive requests. One endpoint seemed to accept the request without that second credential.

We opened an investigation and held back the finding.

We trusted the result only after running a controlled request, observing the security effect, checking a negative control, comparing it with an endpoint that rejected requests missing the same header, and repeating the test with a fresh credential.

The agent pointed us toward the weakness. The test results decided whether it belonged in the report.

How the authorization finding was confirmed

Seeing an endpoint accept an object identifier is not enough to confirm an authorization flaw. We needed to show that it returned or modified a non-public object without the expected ownership check.

The mobile client gave us a useful lead. It sent an application-level credential to the backend and added a separate account-specific credential to sensitive requests. One endpoint appeared not to require the second value.

We then ran five checks in an authorized environment:

1. We built a protocol-valid request with the application credential present and deliberately omitted the account-specific credential.
2. A known-valid account identifier from the authorized test returned account fields, including name and account metadata. The values are redacted in the report.
3. An invalid identifier returned the non-existent-account baseline rather than data.
4. A separate sensitive endpoint rejected a request when the same account credential was omitted, providing a useful comparison.
5. The behavior repeated with a fresh application credential, reducing the chance that a cached response or transient session explained the result.

The first report excerpt shows the application-level bearer, the omitted account-specific credential, and the redacted account fields returned in the response.

The second excerpt records the invalid-account baseline and the comparison endpoint's rejection of the missing account-specific header.

These screenshots come from the report rather than an independent packet capture. They make the investigation traceable. Reproduction still requires another tester to replay the sequence in an authorized environment.

The reconstruction omits customer identifiers, endpoint names, request formats, and response fields.

Together, the checks supported a narrow conclusion: this endpoint returned account data without the expected account-specific credential. The request was not credential-free because it still carried an application-level bearer token. We kept the report at that level instead of calling the path fully unauthenticated.

The report closed the loop with remediation and retest criteria. The backend should enforce ownership, reject requests missing the account credential, avoid treating a client-distributed application credential as user authorization, and normalize account lookup errors. After the fix, the same positive and negative controls should run again.

What changed: proof came before severity

Earlier versions of the workflow could jump from a clue to a conclusion. A response difference could be mislabeled as “injection.” A dangerous browser sink could be called “exploitable XSS.” A redirect could be reported as “authentication bypass.” Each clue deserved investigation, but none had earned the label.

Telling the model to “avoid false positives” did not solve the problem. Instead, we defined the minimum evidence required to confirm each vulnerability class.

Each vulnerability class requires a different observable effect before confirmation.

Candidates that fail these checks stay marked as hypotheses. That filter reduces the chance of an unsupported lead reaching the final report. We do not claim a universal percentage because the rate changes with the target, access level, vulnerability class, and whether early leads are counted alongside report-ready findings.

A useful operational measure is how many reported findings survive reproduction, impact review, and evidence checks without forcing another tester to reconstruct the investigation from scratch.

The harness, not the model, decides what counts

A language model can choose a useful next test. It should not decide on its own when a suspicion becomes a confirmed finding. The surrounding harness enforces scope, controls tools, preserves observations, and checks whether the required proof exists.

Scope enforcement, tool controls, evidence capture, and human review operate outside the model.

Consider server-side request forgery. A user-controlled URL field is a lead. Confirmation requires evidence that the application server, rather than the tester's browser, connected to a controlled destination. The report should preserve the request, callback, build, role, endpoint, and time. It should claim internal-network access only when a separate test demonstrates that impact.

If the evidence remains insufficient, the harness requests another test or records the result as inconclusive. Confidence, repetition, and a detailed explanation cannot upgrade it.

Why we test it on our own systems

We also run the workflow in selected, authorized environments operated by Ostorlab. This provides evidence for internal security and product validation, but it does not replace independent assessment.

That access creates a useful feedback loop. Engineers can compare a claim with the implementation, check whether the agent reached the intended layer, inspect what it missed, and retest a fix under known conditions. Unsupported candidates become negative test cases. When the agent misses a precondition, we add that context to the harness. After remediation, a validated exploit can become a regression test.

This work tests the workflow against known authentication states, implementation details, deployment constraints, and remediation cycles. The results tell us how it performed in those environments. They do not establish a universal accuracy rate.

Applying the same rule to GoPhish

Ostorlab's source-code assessment of GoPhish applied the same evidence rule to a full repository.

Agentic Deep Scan surfaced repository-wide patterns: create paths that could update existing objects, credential checks detached from account state, unsafe browser-rendering paths, and outbound-request controls whose behavior changed with configuration.

Those signals told the team where to look next. They did not automatically enter the report.

The final assessment contained eight report-level findings. Each was traced across the relevant handler, model, middleware, and browser or network path, then paired with a reproducible local proof of concept and remediation guidance. The value came from the complete paths that survived validation, rather than the raw number of patterns detected.

What external research adds

Our first-party cases show how we validate results. Independent research helps answer a different question: how capable are the agents?

In one controlled study, researchers compared ten security professionals, six existing AI agents, and a new multi-agent system called ARTEMIS on a live university network spanning roughly 8,000 hosts across 12 subnets. The stronger of two ARTEMIS configurations placed second overall: nine of its eleven submissions, or 82%, were judged valid, and it outscored nine of the ten professionals under the study's scoring framework.

The result needs context. Participants had up to ten active hours over four days rather than a normal one- or two-week engagement. The environment lacked authentic defensive pressure, the sample was small, and the paper is an arXiv preprint. ARTEMIS also produced more false positives than the human participants and struggled with graphical interfaces.

The study demonstrates capability in one controlled environment. It does not establish equivalence across every application, business workflow, or production constraint. Depending on its harness, an agent may map assets and routes, maintain sessions, compare roles, trace data flows, generate payloads, drive security tools, and adapt after a failed test.

Put humans at the gates, not behind every click

A security practitioner interviewed for this article said AI is already useful for reconnaissance, payload generation, and exploit execution. In their view, people must still own business logic, ambiguous impact, and final validation. They emphasized traceability and reproducibility.

Human approval is not required for every request. It matters when an action could change data, expose customers, or exceed the authorized scope.

Before the test, a human defines the target, build, accounts, data classification, rate limits, and prohibited actions. The execution layer, rather than a sentence in a prompt, must enforce those limits.

During the test, the agent handles high-volume reconnaissance and safe hypothesis testing. A human approves destructive actions, production changes, material privilege escalation, and steps that could expose real customer data.

After the test, a reviewer reproduces material findings, challenges severity and business impact, checks coverage gaps, and accepts or rejects each reportable result. A second model can help with triage, but it is not independent evidence if it merely reads the first model's explanation.

PwC describes a similar workflow: agents perform reconnaissance, humans validate recommendations and information-gathering approaches, and the system proposes targets for testers to investigate.

CREST's research involving 62 cybersecurity providers across 19 countries likewise found AI use concentrated in reconnaissance, analysis, and reporting, with people more involved in higher-risk testing.

Human review helps only when the reviewer has the raw evidence, relevant expertise, and enough time to challenge the conclusion.

Sensitive data is part of the threat model

A pentest may expose source code, architecture, administrator sessions, API tokens, internal URLs, customer records, and working exploits. “We do not train on your prompts” answers only one part of the risk.

Before granting an agent access, a security team should know:

• Whether raw code, traffic, or secrets are sent to an external model API.
• Which prompts, tool outputs, and traces are retained, and who can access them.
• Whether secrets are redacted before model calls and logs.
• Whether execution is isolated and outbound connections are restricted.
• Where data is processed, how long it remains, and how deletion is verified.

The model provider may retain nothing while an orchestration service keeps every HTTP response. The review therefore has to cover the complete data path, not only the model API.

Measure cost per validated result

Reduced reconnaissance effort does not prove lower total cost for every engagement.

To evaluate cost per validated result, include platform fees, model usage, infrastructure, retries, triage, human validation, and governance.

Useful measures include cost per validated material finding, human review time, authorized surface tested per release, rejected or downgraded candidate rate, and time from remediation to verified retest.

To make the business case, compare like-for-like scope and quality. Then measure whether the workflow increases testing frequency or coverage while reducing the expert effort required for validated results.

Would an auditor or customer accept the report?

There is no universal acceptance rule for an “AI pentest.” Acceptance depends on the applicable audit criteria or customer contract and on whether the engagement meets its requirements for scope, methodology, tester qualifications, independence, evidence, human review, and accountability.

Confirm those requirements with the auditor or customer before testing. AI may perform substantial technical work, but it cannot determine whether its own report satisfies those requirements. Framework-specific acceptance is a separate question and should not be inferred from model capability.

So, can you trust an AI pentest result?

Yes, when the evidence can answer six questions:

1. Was the target explicitly authorized, and what remained untested?
2. Which actions ran, and what did the target return?
3. What observable effect proved the vulnerability?
4. Which negative and comparative controls ruled out simpler explanations?
5. Can another tester reproduce the result?
6. Who reviewed the evidence and remains accountable for the conclusion?

The mobile case and the GoPhish assessment reached the same point: only claims supported by reproducible evidence entered the reports.

AI can run the attack.

Trust begins when someone else can inspect it, reproduce it, and reach the same conclusion.

FAQ

What is AI pentesting?

AI pentesting uses AI agents, often based on language models, connected to security tools to investigate an authorized target. Unlike a conventional scanner, an agent can choose the next test from previous observations, maintain sessions, generate payloads, and validate hypotheses. The result is useful only when the control layer records scope, actions, evidence, and limitations.

Can AI replace human penetration testers?

Not reliably for a complete assessment today. AI can automate reconnaissance, known-vulnerability testing, payload generation, and fix retesting. People remain necessary to understand business intent, assess ambiguous impact, approve risky actions, examine coverage gaps, and remain accountable for the final report. AI may replace individual tasks without replacing the complete human role.

Is AI pentesting safe or trustworthy?

Only with strong controls. Execution should remain within an authorized scope, risky actions should require approval, sensitive data should be protected, and humans should validate material findings. Teams must also be able to inspect what ran, what happened, and what remained untested.

Why do AI pentesting tools generate so many false positives?

AI pentesting tools can produce false positives when they mistake suspicious code or unusual responses for successful exploitation. Common causes include missing application context, incorrect assumptions about identity or prerequisites, redirects interpreted as successful authentication, and incomplete runtime access. Runtime validation and reproducibility checks help keep unsupported hypotheses from becoming reportable findings.

What is the difference between an AI finding a vulnerability and proving it?

Flagging a potential vulnerability means identifying a plausible weakness, such as a user-controlled URL reaching a server-side request function. Proving it requires an authorized test that produces an observable effect, preserves the request and response or callback, documents prerequisites, and can be independently reproduced. A hypothesis becomes a confirmed finding only after that validation.

Does AI pentesting put sensitive data at risk?

It can. An AI pentest may process source code, credentials, HTTP traffic, internal URLs, customer records, and working exploits. Teams should verify what reaches external model APIs, what the orchestration platform logs, who can access those records, where data is stored, how long it is retained, and how deletion is confirmed.

Is an AI-generated pentest report acceptable to auditors or customers?

There is no universal answer. An AI-assisted report is acceptable only if it satisfies the applicable audit criteria or customer requirements. Confirm scope, methodology, tester qualifications, independence, AI disclosure, evidence, human review, and accountability in advance. AI can perform technical work, but it cannot determine its own acceptance.

What should stay under human control in an AI-driven pentest?

Humans should control scope, credentials, data classification, prohibited actions, and approval for destructive or production-impacting tests. They should also review material findings, challenge severity and business impact, examine untested areas, and decide what enters the final report. The execution layer should enforce these decisions instead of relying only on prompt instructions.

Does AI pentesting actually save money?

Sometimes. Savings appear when the reduction in repetitive work exceeds the cost of platform usage, infrastructure, retries, triage, human validation, and governance. Compare the complete cost per validated result for equivalent scope and quality, including the effort spent rejecting or reconstructing unsupported findings.

References

• Comparing AI Agents to Cybersecurity Professionals in Real-World Penetration Testing
• Ostorlab: GoPhish Source-Code Assessment with Agentic Deep Scan
• CREST: How AI Is Changing Penetration Testing
• PwC: Multi-Agent Penetration Testing for Broader Coverage