Building a Security Software Factory at Ostorlab

Fully agentic, AI-powered detection will not replace rule-based detection.

That is our conclusion at Ostorlab after building both. An agentic scanner can choose investigative steps, use tools to test a hypothesis, and adapt its next action to what it finds. Those capabilities are powerful, and we have spent the past two years building around them.

We started that work in late 2024. It produced Neutron, which achieved a 97.1% verified exploit solve rate on CyberGym: working demonstrations for 1,463 of 1,507 tasks, reproducing the vulnerability on the vulnerable build without triggering it on the patched build. CyberGym measures the reproduction of known vulnerabilities.

Separately, our agentic security work has led to more than 70 CVE reports. CVE identifiers are the common reference numbers used to track publicly disclosed vulnerabilities. Of our reports, 21 are public at the time of writing, while the remainder await fixes.

We believe deeply in the capabilities of agentic scanning. We also pay its bills.

Deep agentic scans are expensive to run. At the depth we want, running one on every pull request—a proposed code change submitted for review—is economically impractical for both us and our customers. It can also be unnecessary: a known issue with a reliable detection method can often be found much faster with ordinary code.

That is the job of rule-based detection: code that checks for a defined vulnerability or a known pattern of unsafe behavior. A rule might examine a version, a configuration, or how an application responds to a carefully chosen request. Once that check has been developed and validated, subsequent scans can execute it without repeating the investigation that produced it.

Both systems have a job. Agentic scanning can investigate problems that require exploration and reasoning. Rule-based scanning can apply established checks repeatedly and efficiently. Maintaining and improving both is essential to the platform we want to build.

But fast, inexpensive execution does not make a rule inexpensive to maintain.

Ostorlab has used rule-based detection for years. We used to chisel those rules by hand: implement a vulnerability check, encounter an exception, refine the logic, repeat. A false positive is a finding that reports a vulnerability that is not actually present. Manual review helped us catch those mistakes and handle gaps in the rules.

Every new vulnerability, unusual configuration, and false-positive report could create more engineering work. Someone had to investigate, construct a test environment, fix the detection, and review the change. Meanwhile, the same team had new engines and products to build.

Human attention is the rarest resource we have at Ostorlab. A detection platform that keeps improving only when an engineer stops everything else to maintain a rule cannot scale the way we need it to.

We needed a way to keep expanding coverage and correcting false positives while preserving our engineers' attention for the work that needed it most. Agentic systems gave us a way to change the economics of that maintenance.

An agent can spend hours understanding a vulnerability and building a reliable detector. Subsequent scans can execute that detector without paying for the same investigation again.

The expensive reasoning becomes an investment in cheaper, repeatable detection.

Making that work consistently means organizing the entire process: how work starts, how an agent investigates it, how the result is tested, and how a change earns its way into the platform. That connects our problem to a broader shift in how software is being built.

A software factory organizes the work of producing software into a repeatable process. A request enters; implementation, testing, review, and delivery follow. Teams have long automated parts of that process: building an application from its source code, running tests after a change, and preparing a release. People still did much of the work between those automated steps.

AI coding agents extend how much of that process can run on its own. An agent combines a language model with tools and a loop: inspect the code, make a change, run a test, read the result, and decide what to do next. Given a task and criteria for success, it can work through several of those steps without a person directing each one.

This is already changing how engineering teams operate. In its February 2026 account, Stripe described Minions, agents that take a task through implementation and automated checks to a pull request. Stripe retained human review. StrongDM described a software factory in which specifications and test scenarios drive agents that write and validate code without humans writing or reviewing that code.

Security is part of this broader shift too. OpenAI's Codex Security investigates vulnerabilities, validates them, and proposes patches. Closer to detection engineering, AWS researchers describe RuleForge, a system that generates rules for detecting malicious web requests and combines validation with feedback to improve them. Automated security work and rule generation already have meaningful precedents.

The application we wanted to explore was a security software factory whose output is the evolving detection capability of a scanner. A newly disclosed vulnerability or a false alarm from a production scan becomes input to engineering work: investigate it, build or correct the detector, test the behavior, and submit the change for review. The next scan can then benefit from that work.

We had not encountered a public account matching the complete maintenance model we wanted for our scanning platform. Our experience is in putting these pieces together: an engine, an agent responsible for maintaining it, production evidence to guide its work, and a review process that challenges its output.

That still leaves a decision about where human judgment belongs. A dark factory, as we use the term here, runs without people inspecting each code change, much as a factory floor might run with the lights off. People define what the system should produce and how success will be assessed, while machines carry out the implementation and validation.

For Ostorlab, that approach was out of the question. We still have to understand and verify what goes into our systems. Our aim was to make the work reaching a human reviewer much more thoroughly investigated, tested, and challenged, so that their attention could go further.

Today, when we build a detection engine, we also build the agent that will maintain it.

The same production model can turn a feature request into a software change, or scan evidence into a reusable detection improvement:

That is where our Ticket Agents come in. A Ticket Agent combines a job, skills that describe how to carry out that job, and tools that let it act on code and test environments. Rules determine when it runs. A completed scan can trigger it. So can a ticket event or a schedule, such as an hourly run. For detection work, we give it the context and capabilities to turn a known CVE or a vulnerability pattern into a tested rule.

The important part is how much work can happen between receiving that task and opening a pull request. An agent can spend hours constructing environments, running vulnerable software, testing fixed configurations, and finding cases that resemble the vulnerability without actually having it. It can keep refining the detection until those distinctions are represented in code and tests.

We sometimes describe the resulting quality as superhuman. What we mean is the depth of validation we can sustain across routine detection work. A human engineer can do this work too. The constraint is giving every rule that much uninterrupted attention while everything else also needs doing.

There is a concrete example in Asteroid pull request #481, created by our ticket agent to detect a public fallback signing secret in Kubero. Its validation used real upstream releases, a configuration with a deployment-specific secret, and several lookalike services. Those lookalikes included services accepting every token, rejecting every token, and exposing a role endpoint without the expected guard. The pull request (PR) records two true positives and five true negatives: two vulnerable cases correctly detected and five cases correctly left unflagged.

That is a much more useful test than checking whether a vulnerable version number appears in a response. It asks whether the detector can distinguish the specific failure from other behaviors that might look similar.

Another example, Asteroid pull request #478, tested an FTS3 SQL injection detector against eleven cases. These covered real vulnerable and fixed releases, an older release with a different endpoint path, and misleading conditions such as errors, blocked requests, and another product returning similar version information. The PR records three true positives and eight true negatives. These are results from curated test environments; the production false-positive rate is a separate measurement.

Generating a convincing implementation is only part of the process. Every proposed change also has to survive review.

Over the past year, we have built and continuously improved our internal PR review bot. It challenges whether a change implements the right behavior and whether the evidence supports what the implementation claims.

The Kubero PR shows why this matters. The review bot caught a flaw in a negative control: a failed request could be treated as evidence that the server had rejected an invalid token. A timeout does not establish that. The review required an explicit rejection before the detector could confirm the finding. That is exactly the kind of distinction that prevents a false positive.

Building the reviewer ourselves lets us adapt it to our code, detection requirements, and recurring mistakes. Its quality has improved enough over successive iterations that a positive review has become a strong signal that a PR is ready for a human merge decision. We still perform high-level checks and architecture review. The amount of detailed human review needed for routine changes has fallen substantially.

This review system is a significant production expense. Roughly 99% of its direct running cost is AI inference and 1% is infrastructure. The value comes from the human attention it frees while improving the quality of the changes reaching us.

The combination of Ticket Agents and automated review changed our unit of engineering work.

Previously, we built an engine and then accumulated a body of detection rules that our team had to maintain. Now we build the engine together with a maintainer. That maintainer works from production runs and their output: addressing false positives, adding rules, and proposing improvements. The review bot challenges its changes, and human judgment remains part of deciding what enters the platform.

The maintenance loop connects production evidence to tested changes, review, and an updated engine:

When that process fails, our engineering effort goes into improving the maintainer.

A bad rule tells us something about the system producing rules. The agent may need better instructions, another tool, a more representative test environment, or a stronger review requirement. Improving those capabilities can help with the next hundred changes, as well as the one that exposed the weakness.

This has a practical consequence for a small engineering team. While we work on substantial efforts such as runtime application self-protection and our large banking security report, the Ticket Agents continue producing new detections and false-positive fixes every day, with very little human intervention in that work.

Maintenance keeps moving while the team takes on projects that need sustained human focus. As we improve the agents and their review process, we expect to expand how much work they can handle.

It has also changed how we think about adopting open-source projects.

We used to see an existing project primarily as engineering work we could bring into our platform. That saved development effort, but we still had to account for integration and ongoing maintenance.

Our preference has shifted toward implementing more internally and pairing each implementation with an agent maintainer from the outset. Owning the implementation gives us room to shape its behavior, tests, and maintenance process around our requirements. The ability to keep improving that implementation changes the economics of the original build decision.

For our users, the benefit should show up in the findings they receive.

A false positive becomes input to a process that can investigate and correct the underlying detection. Sometimes the correction is a tiny condition. Sometimes it takes substantial logic to handle an unusual configuration. The work is to keep chiseling the rule until it reflects the behavior we actually need to detect, including the awkward exceptions.

Coverage can evolve through the same process. Agents can keep identifying missing detections and building them while our engineers work elsewhere. Each accepted improvement becomes reusable across subsequent scans.

Over the past month, our internal measurements show:

The charts index each previous level to 100. They show the reported ratios, not a daily trend. The current false-positive rate is one hundred-and-twentieth of the previous rate; its index is approximately 0.83, not an absolute false-positive percentage.

These are results from our own use of the system. A small number of customers are adopting Ticket Agents, and the same capabilities we use internally are available through the Ticket Agent configuration: define the job, provide the skills and tools, and choose the events that should start the work. The results will depend on the task and the system built around it.

The larger opportunity is what this lets our engineers attempt next. We can take on engines and capabilities whose ongoing detection work would previously have exceeded our capacity to maintain them. We build the foundation and the maintainer together, then improve both from their behavior in production.

Human attention remains essential to our security factory. We are putting it where an hour of engineering can improve thousands of future decisions.