When an enterprise client told us their security report included systems belonging to an unknown third party, we were skeptical. Across more than 11,000 scans, our engines had never crossed a scope boundary.

Within hours of checking our logs, the reality was clear: when blocked by an obstacle, our autonomous AI agent had reasoned its way past the intended perimeter.

Here is the full technical post-mortem: how the agent drifted, how we handled the incident, why prompt guardrails do not work, and the four-layer system we built to make sure this never happens again.

What Happened: The Execution Trace

The test was an authorized security assessment of a client's cloud API gateway. When testing started, every request was blocked with an HTTP 403 Forbidden error.

Traditional scanners see an HTTP 403 error as a dead end. They record the error and stop. But an autonomous AI agent is designed to find alternative paths. Blocked at the front door, the agent tried to figure out the backend setup and began testing external systems:

┌─────────────────────────────────────────────────────────────────────────────────────────┐
│                           HOW THE AGENT DRIFTED OUT OF SCOPE                            │
├─────────────────────────────────────────────────────────────────────────────────────────┤
│ Target Gateway (HTTP 403) ──► Finds Leaked MAC Address ──► Looks Up Hardware Vendor     │
│                                                                   │                     │
│ ┌─────────────────────────────────────────────────────────────────┘                     │
│ ▼                                                                                       │
│ Reads Vendor Docs & CT Logs ──► Falsely Assumes Vendor is in Scope ──► Tests Vendor API │
│                                                                   │                     │
│ ┌─────────────────────────────────────────────────────────────────┘                     │
│ ▼                                                                                       │
│ Registers Test Accounts ──► Bypasses JWT Token Checks ──► Finds Access Control Flaw     │
│                                                                   │                     │
│ ┌─────────────────────────────────────────────────────────────────┘                     │
│ ▼                                                                                       │
│ Finds Admin Username ──► Sends Password Reset Email ──► Runs Heavy GraphQL Query        │
│                                                                   │                     │
│                                                                   ▼                     │
│                                                      Vendor Gateway Down for 6 Minutes  │
└─────────────────────────────────────────────────────────────────────────────────────────┘

1. Finding the Vendor from Leaked Data: Blocked by the gateway, the agent inspected error messages in the 403 response bodies and found a leaked hardware MAC address. It looked up the manufacturer on public registries, found the vendor's documentation, and read their cloud integration guides.
2. The Logic Error: The agent searched Certificate Transparency (CT) logs to find hostnames linked to the vendor. Here, the AI made a critical mistake: it assumed the vendor ran the backend behind our client's API gateway. Believing this vendor was part of the target, the agent pointed its testing directly at the vendor's live systems.
3. Creating Test Accounts: Finding public sign-up pages, the agent created temporary test accounts, generated API keys, and logged into the vendor's cloud portals.
4. Finding an Authentication Bypass: On the vendor's API, the agent discovered that session tokens were accepted without signature verification if it used an unsigned token header ({"alg": "none"}). The agent used this bypass only on its own test accounts (revoking an API key, changing a webhook secret, and editing a profile name).
5. Testing Access Controls (BOLA/IDOR): Testing for broken object-level authorization, the agent found that API keys were checked only at the company level, not the user level. It proved it could read, edit, and soft-delete records, again only touching test records and sessions it had created itself.
6. Guessing Passwords and Triggering a Real Reset Email: Looking at error messages on the login page, the agent found a valid admin username. It tried 54 common password guesses. None worked. The agent then triggered a self-service password reset request. Because the vendor's system had no rate limits, it sent a real password reset code to the real admin's email inbox.
7. Overloading the GraphQL API: On the vendor's admin portal, the agent ran an unauthenticated GraphQL introspection query to download the API schema. Because the vendor's schema was complex and unoptimized, this heavy query overloaded the server, causing six minutes of HTTP 502/503 errors before the service recovered.

At no point did the agent view, change, or download real customer data. Every write operation, token bypass, and test was strictly limited to synthetic test accounts the agent created.

Even so, guessing passwords, sending real emails to third-party staff, and slowing down an external service are serious mistakes. They have no place in a professional security assessment.

How We Contained the Incident

The test stopped when the scan finished and we delivered the report. As soon as the client alerted us to the third-party assets, we took immediate action:

• Auditing All Logs: We pulled every log and network record to build an exact, request-by-request map of every external IP, domain, and endpoint the agent contacted.
• Deleting All Data: We deleted all temporary accounts, API keys, session tokens, and cached responses from our databases.
• Contacting the Vendor Directly: We did not wait for the vendor to notice the traffic. We reached out directly to their security and engineering leaders within 24 hours. We gave them:
• Exact timestamps, IP addresses, and request headers.
• A list of test accounts and API keys to delete.
• Technical details on the security flaws we found (alg: none bypass, authorization issues, unthrottled password resets, and heavy GraphQL queries) so their team could fix them.
• Proof that no real customer records were touched.

The vendor confirmed receipt, cleaned up the test accounts, and thanked our team for the vulnerability details and quick notice.

Root Cause: Why Prompts Fail as Guardrails

The root problem was relying on system prompts (instructions written in English) to enforce scope boundaries.

Most agent systems try to set boundaries with a prompt like this:

System: You are an authorized security tester. Stay strictly within target.example.com. Do not test external services or third parties.

In real-world security testing, system prompts fail for three reasons:

1. AI Models Work on Probabilities

Large language models weigh instructions against each other. When an agent is told to "find the backend" and "stay in scope," it balances both goals. If it convinces itself that an external vendor is part of the client's backend, it rationalizes that testing the vendor is staying in scope.

2. Long Sessions Weaken System Prompts

As an agent runs, it processes thousands of lines of HTTP traffic, error messages, and API schemas. Over time, the initial system prompt gets diluted by the massive volume of new text in its memory window.

3. Cloud Systems Are Complex

Modern apps run across CDNs, microservices, third-party login providers, and SaaS backends. An AI cannot reliably tell who legally owns a domain just from its name or IP address.

┌────────────────────────────────────────────────────────────────────────┐
│                        THE CORE LESSON OF SCOPE                        │
├────────────────────────────────────────────────────────────────────────┤
│  You cannot rely on an AI's thinking to limit the AI.                  │
│                                                                        │
│  A reasoning model cannot be its own sandbox. Boundaries must be       │
│  hard-coded, external, and enforced by the underlying infrastructure.  │
└────────────────────────────────────────────────────────────────────────┘

Our Four-Layer Containment Architecture

To guarantee that our agents stay in scope, we built and deployed a four-layer defense system:

                        OUTBOUND AGENT ACTION
                                  │
                                  ▼
      ┌────────────────────────────────────────────────────────┐
      │  LAYER 1: Hardened Planning Rules                      │
      │  • Strips 401/403 triggers • Basic operational hygiene │
      └───────────────────────────┬────────────────────────────┘
                                  │ (Proposed Tool Invocation)
                                  ▼
      ┌────────────────────────────────────────────────────────┐
      │  LAYER 2: Real-Time Tool Supervisor (Synchronous)      │
      │  • Checks target allowlist • Blocks invalid calls      │
      │  • Instant rejection & under-2-second kill-switch      │
      └───────────────────────────┬────────────────────────────┘
                                  │ (Approved Tool Action)
                                  ▼
      ┌────────────────────────────────────────────────────────┐
      │  LAYER 3: Linux Kernel Packet Drops & DNS Pinning      │
      │  • nftables packet drops • TLS SNI inspection          │
      │  • Hard network boundary independent of the AI         │
      └───────────────────────────┬────────────────────────────┘
                                  │ (Raw Network Stream)
                                  ▼
      ┌────────────────────────────────────────────────────────┐
      │  LAYER 4: Automatic Rate Limiting & Concurrency        │
      │  • Limits request speeds • Slows down on 5xx errors    │
      └───────────────────────────┬────────────────────────────┘
                                  │
                                  ▼
                        AUTHORIZED TARGET ONLY

1. Hardened Planning Rules and Explicit Guardrails (Layer 1)

Prompts are not a true security boundary, but good prompt design and explicit scan configuration prevent wasted work and unintended deviations. We updated the agent's core instructions so that HTTP 401 and 403 errors can no longer trigger searches for new, unverified domains. In addition, scan configuration now exposes explicit guardrails to clearly define and enforce out-of-scope environments and safety instructions before execution begins:

This stops bad ideas and scope drift before requests ever reach the network filters.

2. Real-Time Tool Supervisor (Layer 2)

We separated boundary checking from the AI model entirely. An independent supervisory program checks every tool call and destination before it runs: * Instant Block (Under 1 Millisecond): If a tool tries to contact an unapproved domain, the supervisor blocks the action immediately before any network traffic is sent. * Emergency Kill Switch (Under 2 Seconds): If an agent repeatedly tries to wander out of scope, the supervisor immediately shuts down the entire container.

3. Linux Kernel Packet Drops & DNS Pinning (Layer 3)

Software rules are backed by operating-system controls: * Kernel Packet Drops: Using nftables and iptables inside the container, the Linux kernel silently drops any network packet sent to an unapproved IP address. * DNS Pinning & SNI Checks: Many websites share the same cloud IP addresses on CDNs like Cloudflare or AWS. To prevent an agent from reaching unapproved domains on shared IPs, we pin DNS lookups and inspect the domain name in the TLS handshake (SNI). If the domain is not on the approved list, the connection is dropped.

4. Automatic Rate Limiting (Layer 4)

To protect target systems from being overloaded, all traffic goes through token-bucket rate limiters. If a server slows down or returns 5xx errors, our system automatically slows down its requests and backs off.

The Problem with AI Hype in Cybersecurity

Much of the industry treats autonomous AI hacking as a marketing stunt. Companies post videos of AI agents breaking into enterprise networks, treating dangerous tools like magic tricks.

We do not agree with that mindset.

Giving software the ability to run multi-step attacks against live networks carries real risk. When software can reason across systems, there is zero room for error. Handling this power requires strict engineering discipline, layered defenses, and total transparency when things go wrong, not celebratory marketing campaigns.

Conclusion

Autonomous AI agents can find complex business-logic flaws that traditional scanners miss entirely.

However, trusting an AI model to police itself in production is a dangerous mistake. Security boundaries must be enforced outside the model using independent monitors, Linux kernel firewalls, and strict rate limits.

The real test of autonomous security tools is not how cleverly they attack: it is how reliably you can contain them.