Ostorlab vs. Manual Pentesting: More Critical Findings, Less Time, Lower Cost
Ostorlab reports more critical findings, shorter testing time, and assessment costs of one-tenth to one-fortieth of manual pentests in completed head-to-head PoCs. Examine the technical evidence and comparison limits.
In one enterprise application assessment, Ostorlab obtained an administrative token without a client secret, used it to enter an integration feature, and executed supplied code inside a server process with local administrator rights. Each step unlocked the next: an identity-service misconfiguration became privileged server-side execution. That is the investigation depth AppSec teams should compare when evaluating Ostorlab alongside a manual pentest.
Our team reports that Ostorlab found more critical vulnerabilities, completed testing sooner, and cost one-tenth to one-fortieth as much as the manual pentests it ran alongside in every completed head-to-head proof of concept. The completed comparisons covered mobile applications, web applications, source code, and Active Directory testing.
Ostorlab vs. manual pentesting: reported PoC results
| Measure | Result reported by the Ostorlab team |
|---|---|
| Critical findings | More critical findings than the parallel manual assessment in every completed comparison. |
| Testing time | Testing completed sooner in every completed comparison; engagement durations and a measured speedup range are not published here. |
| Assessment cost | One-tenth to one-fortieth of the parallel manual assessment cost across the completed comparisons. |
These are our team's engagement results for the scopes and access conditions assessed. They are not an independent benchmark or a guarantee for another application. This article does not publish the cohort size, matched finding counts, engagement durations, or itemized cost breakdown. The reported cost range is therefore not a forecast of your total savings; a comparison on your scope should include setup, human review, retesting, and engineering effort.
Three attack chains, from entry point to impact
The following anonymized examples come from Ostorlab finding records and execution histories. They show what the platform investigated and confirmed. They do not identify findings missed by a particular human tester or independently substantiate the aggregate comparison results; customer-level scorecards and auditor reports are not published here.
| Investigation | Demonstrated result | Limit of the evidence |
|---|---|---|
| Secretless token → server execution | Protected API access, supplied code execution, privileged process identity, and outbound connectivity | Results apply to the assessed configuration and access path. |
| Ordinary domain user → CA account → certificate → domain replication | Recorded lab chain reached krbtgt replication and golden-ticket forgery |
Separate lab; the issued administrator certificate remained unrevoked. |
| Mobile credential → Management API | Token issuance for a second audience and a directory-page read | User-management write scopes were present but not exercised. |
The enterprise and mobile cases below are customer assessments. The Active Directory case is a separate lab assessment, not customer comparison evidence. Each chain identifies how one weakness enabled the next step and where validation stopped. Code excerpts use synthetic values to illustrate the reported mechanisms.
To inspect the deliverable before arranging a call, open the web sample report or mobile sample report.
Case 1: no client secret → admin token → privileged server execution
Attack chain: Client identifier without a secret → administrative token → authenticated integration feature → supplied C# executes in the IIS worker → local administrator rights and outbound connectivity confirmed.
Stage 1: obtain a usable administrative token. Ostorlab submitted a token request using a client identifier and an administrative scope, with no client secret. The identity service returned a bearer token. Ostorlab then inspected the returned token's claims and used it against a protected directory operation, which returned organizational profiles.
Stage 2: use that token to execute server-side code. In the same enterprise application assessment, Ostorlab used the administrative token obtained without a client secret to reach an authenticated integration feature in a vendor-supplied IAM module. The feature accepted custom C# and compiled it inside the application server process. Controlled timing tests, subprocess execution, host diagnostics, and outbound network checks demonstrated execution under the application's IIS app-pool identity — an account that was a member of BUILTIN\Administrators and held SeDebugPrivilege and SeImpersonatePrivilege.
The agent worked through the feature's expected interface. It wrote a probe that implemented that interface, so the payload passed compilation and type checks before its constructor ran. The application used Roslyn to compile the source in-process, loaded the resulting assembly with AssemblyLoadContext.Default.LoadFromStream(), and created the instance with Activator.CreateInstance().
Stage 3: confirm execution, privileges, and outbound access. A normal request established a baseline. Constructor payloads introduced controlled delays. The recorded response times followed the requested progression: a baseline around 122 ms, then roughly 2.13 s, 5.14 s, and 10.15 s for the two-, five-, and ten-second probes, with repeated probes following the same pattern. Further tests recorded subprocess execution, the identity running the code, and outbound DNS and TCP connections to a tester-controlled host.
The integration feature compiled and instantiated supplied C# in its server process. The requested delay ladder supported constructor execution; separate subprocess tests, host diagnostics, and egress checks established additional impact.
| Test stage | Recorded observation | Supported conclusion |
|---|---|---|
| Baseline integration input | Normal response time, around 122 ms | A reference for subsequent measurements. |
| Constructor delays of two, five, and ten seconds | Response times of roughly 2.13 s, 5.14 s, and 10.15 s; repeated probes followed the same pattern | Supplied constructor behavior executed synchronously. |
| Subprocess test | Process execution evidence | Execution extended beyond merely parsing or compiling a file. |
| Host diagnostics | The worker ran as the IIS app-pool identity, was a member of BUILTIN\Administrators, and held SeDebugPrivilege and SeImpersonatePrivilege |
The code inherited administrative rights and credential-theft primitives, not just application-level execution. |
| Outbound network test | DNS resolution and TCP connections to a tester-controlled host succeeded | The same process could reach external infrastructure for exfiltration or payload retrieval. |
The negative control. The agent also tested a sibling integration endpoint that enforced pre-instantiation type gating. The same probe was rejected before compilation or instantiation, and no timing effect appeared. This control test showed that the two integration endpoints applied different validation before executing supplied code.
Assessment result. The timing tests demonstrated constructor execution. Separate subprocess, identity, and egress tests established execution privileges and network access. The integration endpoint required authentication, supplied by the token obtained at the start of this chain. The demonstrated sequence was secretless token issuance, administrative API access, and authenticated server-side code execution.
The remediation direction. Review the client's authentication requirements and permitted grant types; restrict its scopes and audiences; shorten the lifetime to match its purpose; and invalidate credentials issued under the vulnerable configuration as appropriate. The retest should cover both new issuance attempts and the treatment of previously issued credentials.
Keep untrusted integrations outside the privileged application process:
- Use a separate worker with an appropriate OS-enforced security boundary. A collectible
AssemblyLoadContextsupports unloading assemblies; it is not a security sandbox. Microsoft describes the distinction between loading isolation and process security boundaries. - Give the worker a low-privilege identity, restricted access to secrets, and an egress allowlist.
- Restrict who may create or modify integrations.
- Retest authorization, process identity, protected-resource access, and disallowed egress. If executing integrations is an intended feature, successful compilation alone is not a failed fix.
What Ostorlab automated. This chain required the agent to carry a credential from one finding into another test, construct a probe compatible with the integration interface, compare timing results, and establish process privileges and network access. This is attack-chain investigation and impact validation within the assessment. The comparison with a manual pentest should assess whether either provider found and demonstrated the complete path, alongside the time and cost required to do so.
Case 2: ordinary domain user → CA account takeover → administrator certificate → domain replication (lab)
Attack chain (lab): Ordinary domain user → ownership of the CA service account → control of its certificate template → administrator certificate → domain replication of krbtgt → golden ticket.
In an Active Directory lab assessment, the agent started with a low-privileged domain user and enumerated the directory's access-control surface. The review covered 3,002 ACL rows across users, groups, computers, and the domain root. It identified a non-inherited WriteOwner ACE that allowed an ordinary user to take ownership of the certificate authority's service account. It was the only ACE in the corpus that gave a non-Tier-0 principal control over a Tier-0 object.
The CA service account was a member of Cert Publishers, held GenericAll on a certificate template, and was Kerberoastable with RC4.
The GenericAll grant on the certificate template was the ESC4 condition: it allowed the agent to modify the template's security settings before requesting a certificate as another principal. The WriteOwner ACE on the service account was the separate entry point into that template-control path. Cert Publishers membership and RC4 Kerberoastability were additional exposure, not prerequisites for the ESC4 step.
The expected boundary. A CA service account and its certificate templates are Tier-0 objects. Ordinary users should not own them, template ACLs should be restricted to the CA and directory administrators, and every issuance should be logged and revocable. A WriteOwner ACE on the CA account is not a routine delegation.
The investigation. The agent seized ownership of the service account, rewrote the DACL to grant itself GenericAll, and controlled the account. It modified the template to allow enrollment, requested a certificate as an administrator, authenticated with that certificate, and replicated krbtgt. It then forged a golden ticket. The misissued administrator certificate remained unrevoked.
One non-inherited ACE gave an ordinary user ownership of the CA service account. The agent chained that ownership through an ESC4 template to a certificate and domain replication.
Illustrative execution pseudocode · synthetic names; not real tool syntax.
# Lab: ordinary user -> CA service account -> template -> certificate -> DCSync.
# Names and values are synthetic.
ace = read_ace(target="svc-ca", trustee="user-a")
# One non-inherited WriteOwner ACE in 3,002 ACL rows.
seize_owner(target="svc-ca", as="user-a")
grant_generic_all(target="svc-ca", to="user-a")
reset_secret(target="svc-ca")
template = modify_template("CorpAuthTemplate", enroll="svc-ca")
cert = request_certificate(template, subject="Administrator")
session = authenticate_with(cert)
replicate_krbtgt(session)
forge_golden_ticket()
# The misissued administrator certificate was not revoked.
| Step | Evidence | Boundary assessed |
|---|---|---|
| Enumerate ACLs across the directory | 3,002 rows; one non-inherited WriteOwner ACE for an ordinary user on the CA service account |
Whether Tier-0 objects were isolated from ordinary users. |
| Seize ownership and rewrite the DACL | Owner seizure and GenericAll grant reproduced live; writes rolled back byte-equal |
Whether the ACE was exploitable, not merely present. |
| Control the CA service account | The account was a member of Cert Publishers and held GenericAll on the template |
Whether the service account's rights reached certificate issuance. |
| Modify the template and request a certificate | ESC4 condition; an administrator certificate was issued | Whether template ACLs prevented enrollment as a privileged principal. |
Authenticate with the certificate and replicate krbtgt |
Domain replication and a golden ticket | Whether certificate-based authentication reached domain-controller authority. |
| Check revocation | The misissued administrator certificate remained unrevoked | Whether the organization could withdraw the credential. |
What the evidence proves. The ACL takeover was reproduced live, and the account's rights reached a template the CA would use. The certificate and replication leg is recorded as the agent's critical chain; the misissued certificate was still unrevoked when the assessment ended. This is a lab assessment, not a customer result or comparative benchmark. The unrevoked certificate is an unresolved cleanup item, not evidence of a completed restoration. An equivalent engagement needs a named owner to revoke it and verify the result before accepting cleanup as complete.
Assessment result. Control of the CA service account provided access to a certificate template that could be modified to request an administrator certificate. The recorded sequence connected the initial ACL misconfiguration to certificate-based authentication and domain replication. The lab did not compare automated and manual assessment performance.
The remediation direction. Remove the ACE and review every non-inherited grant on Tier-0 objects. Separate the CA service account from ordinary administration, restrict template ACLs to the CA and directory administrators, disable RC4 for service accounts, require approval and logging for template changes and certificate issuance, revoke misissued certificates, and rotate krbtgt twice. The retest should verify that a low-privileged user cannot own the CA account or modify the template, that DCSync fails, and that forged tickets are rejected after rotation.
An internal-testing report should document the starting identity, affected ACLs, permissions obtained, validation results, unsuccessful tests, and remaining artifacts. Administrators need this information to reproduce the finding, repair the delegation, and verify cleanup within the authorized scope.
Case 3: mobile binary → live credential → broader API audience → tenant directory access
Attack chain: Shipped Flutter iOS binary → extracted client credentials → valid token for the app’s API → reuse against 38 candidate audiences → Management API token → confirmed read of tenant user records.
Ostorlab assessed an iOS application built with Flutter that used Auth0 machine-to-machine credentials to obtain API access tokens. The application's compiled binary contained the client_id and client_secret, supplied at build time through Flutter's DART_DEFINES mechanism. The assessment tested whether those credentials were valid and which APIs they could access.
The expected boundary. A machine-to-machine credential should be scoped to the one audience and set of operations its service needs. A credential embedded in a distributed client is not a secret; it is available to anyone who can obtain the build. The assessment therefore examined both whether the credential was live and what else it could authenticate to.
The investigation. The agent extracted the credential from the Mach-O binary and requested a token from the tenant's /oauth/token endpoint. The request succeeded with HTTP 200 and a signed JWT; a fabricated credential returned HTTP 401. The token carried a read:TSC scope, a client-credentials grant, and a signing key traceable to the tenant's JWKS endpoint.
To assess access beyond the API used by the app, Ostorlab submitted token requests for 38 candidate audiences to the same token endpoint. In these requests, the audience identifies the API for which access is requested. A token request for the tenant's Management API audience returned HTTP 200 with a token carrying eight scopes, including update:users, delete:users, create:users, and create:client_credentials. A single non-destructive GET to the Management API returned the first page of a 1,000-record user directory containing emails, phone numbers, last IPs, and login behavior. The assessment demonstrated this read; it did not exercise the token's user-create, update, or delete permissions.
The token endpoint accepted the extracted client credentials for both the internal-service audience and the Management API audience. A subsequent directory request confirmed read access through the Management API.
Illustrative HTTP sequence · synthetic host, client, and token values.
# The credential is extracted from the mobile binary.
POST /oauth/token HTTP/1.1
Host: tenant.example.invalid
Content-Type: application/x-www-form-urlencoded
grant_type=client_credentials&client_id=MOBILE_CLIENT&client_secret=EXTRACTED_SECRET&audience=AUDIENCE_A
# Article outcome: a read:TSC token for the internal service.
POST /oauth/token HTTP/1.1
Host: tenant.example.invalid
Content-Type: application/x-www-form-urlencoded
grant_type=client_credentials&client_id=MOBILE_CLIENT&client_secret=EXTRACTED_SECRET&audience=MANAGEMENT_API
# Article outcome: a token with eight management scopes.
GET /api/v2/users?per_page=100 HTTP/1.1
Host: tenant.example.invalid
Authorization: Bearer FAKE_MANAGEMENT_TOKEN
# Article outcome: the first page of a 1,000-record directory.
| Evidence collected | What it established |
|---|---|
| Credential extracted from the shipped iOS binary | The secret was distributed to every installation, not held server-side. |
| Token request with the extracted credential returned HTTP 200; a fabricated credential returned HTTP 401 | The credential was live, not a stale build artifact. |
read:TSC scope and a client-credentials grant in the first token |
The app's intended audience was an internal service. |
| 38 candidate audiences tested against the same token endpoint | The credential's authorization surface was broader than the app's use. |
| Token request for the Management API audience returned a token with eight scopes, including user create, update, and delete | The token carried management write permissions; those operations were not exercised. |
| One non-destructive GET returned the first page of a 1,000-record user directory | The impact extended to tenant-wide PII, not just a scope label. |
What Ostorlab automated. This assessment combined binary analysis, live credential validation, audience enumeration, and a controlled API read. A tester performing the investigation manually would need to carry out and connect those same steps. Ostorlab established the credential's usable access beyond the API called by the app, giving the AppSec team a validated exposure to prioritize and a broader retest requirement.
The remediation direction. Remove the secret from the client and rotate it. Review the client's permitted audiences and scopes, and separate tenant-management credentials from client-facing service credentials. Invalidate tokens issued under the vulnerable configuration. The retest should enumerate the client's full authorization surface, not only the audience the app calls.
A longer write-up of this case is available in the Agentic Deep Scan case study.
Running Ostorlab alongside a manual pentest
Give Ostorlab and the manual pentest provider equivalent targets, accounts, access, and permitted actions. This allows your team to compare the investigation and validated findings under the same conditions. Start with a representative staging environment and dedicated test accounts where practical. If production testing is required, agree permitted actions, data access, traffic limits, and incident escalation with the application owner before execution. A connection discovered in an app does not automatically authorize testing its identity provider or analytics service.
Use this operating plan when scoping the evaluation:
| Decision | What to agree before execution |
|---|---|
| Assets and access | Customer application owner identifies builds, endpoints, accounts, roles, supporting services, and exclusions; both providers receive equivalent access for a comparison. |
| Enforced restrictions | Assessment operator configures firewall exclusions and traffic limits, verifies them, and records which scope restrictions are guidance rather than enforced network boundaries. |
| Supervision and stopping | Name an operator and a customer incident contact. Agree how to stop the scan, confirm testing has ceased, revoke test access if needed, and escalate unexpected effects. Verify the stop procedure before production testing. |
| Schedule | Record the timezone, permitted windows, scan pause behavior, and final stop conditions. Choose a scan type and operating procedure compatible with any strict cutoff. |
| Changes and cleanup | Assign an owner for each test account, session, file, credential, and configuration change. Record restoration or revocation evidence and have the customer owner accept the handoff. |
These are engagement decisions to confirm with the assessment team, not a promise that every action is automatically approved or reversed.
Scope and action controls
The October platform update describes scope guidance, firewall exclusions, traffic controls, monitoring, and requested-action approvals. The scope whitelist guides the agent; it is not a hard network boundary. Firewall exclusions block configured addresses and networks. Monitoring adds drift detection and optional shutdown.

Approval settings govern decisions requested by the agent. “Ask me, block if no answer” keeps a requested action waiting for a decision; it does not require a human review before every tool call. The operator must configure these settings for the engagement.

Execution windows and stop conditions
Execution windows specify days, hours, and timezone. A scan created outside its window waits; a pausable scan pauses at closing and resumes in the next window. Non-pausable scans continue until completion after closing. Confirm the chosen scan type's behavior before treating a closing time as an operational cutoff. A recurring schedule also needs a separate final stop condition.

See the execution-window documentation for the supported scheduling behavior. These screenshots are platform examples from public documentation, not evidence captured from the anonymized assessments.
Compare the investigation evidence and handoff
A findings count is only useful if your team can verify the findings. Compare the manual provider's supporting evidence with Ostorlab's investigation record, including how each established exploitability and impact. Ostorlab's AI Pentest analysis view connects plans and tasks to recorded tool names, arguments, outputs, and timestamps. Inspect an important finding from its initial hypothesis through the response establishing impact. A source trace, a successful runtime test, and a blocked attack stage should be distinguishable in the report.


The handoff should identify starting privileges, reproduced impact, unsuccessful branches, outstanding checks, and the proposed repair. Count distinct validated findings, with duplicate root causes and attack chains reconciled. Do not count an untested hypothesis as a confirmed critical vulnerability.
Execution records also help the operator verify cleanup. One reviewed web assessment recorded a support session being created, checked, revoked, and checked again to confirm logout. Apply the same discipline to changed configuration, uploaded files, and issued credentials. Record unresolved artifacts explicitly and assign an owner; logging an action is not the same as reversing it.
Extend the comparison through remediation
The comparison should include the work required to turn a finding into a verified repair. Ostorlab can carry a finding into remediation with custom fix recommendations and proposed code changes delivered through supported repository integrations. Developers review the resulting diff in their existing pull-request workflow, run the relevant checks, and use follow-up testing to assess whether the changed application resolves the original issue.
Retest the complete chain after repair. Correcting token issuance must also block downstream access with previously issued credentials. Removing an embedded mobile secret must also remove its access to the Management API.
The table lists remediation and retest criteria derived from the cases; it does not claim that any particular patch has been created or merged:
| Finding | Proposed repair direction | Meaningful retest |
|---|---|---|
| Secretless administrative token issuance | Correct client authentication, grants, scopes, audience, and credential lifecycle. | Unauthorized issuance fails; downstream access follows the corrected policy. |
| Authenticated code execution in an integration feature | Isolate untrusted execution in a separate restricted worker; enforce authorization, least privilege, and egress policy. | Unauthorized execution fails; permitted integrations cannot access protected resources or disallowed network destinations. |
| AD ownership and certificate-template chain (lab) | Remove the ownership grant, restrict template control, revoke the issued certificate, and rotate domain credentials. | A low-privileged user cannot control the CA account or template; issued credentials and forged tickets no longer work. |
| M2M credential with tenant-management audience | Remove the secret from the client; rotate; restrict audiences and scopes; separate management credentials. | The client cannot obtain a Management API token; previously issued tokens are invalid. |
- Ostorlab Autofix supports custom fix recommendations, changes routed through pull requests, and follow-up verification workflows.
- Source-code testing keeps affected paths and code revisions connected to remediation.
A generated patch still goes through engineering review and appropriate testing. Follow-up assessment evidence informs closure; a proposed diff or a successful build alone does not establish that an exploitable path has been eliminated.
Compare this workflow with the remediation and retest service included in your manual pentest. Record whether developers can reproduce the finding, implement the proposed repair, and verify that the original access or execution path is blocked, along with the effort required from your team.
Measure Ostorlab against your manual pentest provider
Use the reported PoC results as the proposition to test: more validated critical findings, shorter testing time, and lower assessment cost. A useful comparison starts with the same build, targets, accounts, source access, permissions, and exclusions. Agree the assessment window and decide how to handle changes made during testing. Record incomplete engagements and limitations rather than dropping them from the results.
| Measure | What to record for each provider |
|---|---|
| Validated findings | Counts by an agreed severity rubric, supporting evidence, overlap, unique findings, and duplicate or disputed results. Have the customer security lead adjudicate disagreements. |
| Delivery time | Start when equivalent access is ready; stop at delivery of the usable report. Record setup time, blocked time, and retest turnaround separately. |
| Total evaluation cost | Provider or platform fees, model usage if separately billed, setup, human validation, reporting, retesting, and customer engineering effort. Use the same labor-rate assumptions for both sides. |
| Remediation usefulness | Whether engineering can reproduce the finding, identify the failing control, and verify the repair; record time spent on that handoff. |
Before authorizing a PoC, request an itemized quote identifying inclusions, usage limits, overage terms, retest allowance, and the customer's preparation work. Agree delivery dates after scope and access are confirmed. Use those terms and actual effort to determine whether the assessment-cost advantage reported in our completed PoCs translates into lower total cost for your team.
Questions to settle before replacing an existing pentest
Where does Ostorlab fit in our testing program? Use an initial parallel evaluation to decide which application-testing activities it can take over on your scope: authenticated workflow testing, investigation across apps and APIs, and repeat testing after changes. Keep human ownership of scope, business intent, disputed findings, operational decisions, and acceptance of fixes. Retain specialist testing for activities outside the agreed coverage. Use the findings, delivery, cost, and assurance requirements to decide how much of your current manual engagement Ostorlab can replace.
Do we have to provide source code? Ostorlab supports mobile builds, web applications, APIs, and source-code testing. The mobile chain began with a shipped binary; the enterprise chain used runtime tests. Agree the assessment mode and access explicitly for a fair comparison.
What does our team need to prepare? Supply the build or target URLs, authorized services, role-separated test accounts, representative test data, and relevant API or workflow documentation. For logged-in flows, arrange the required MFA or OTP test setup with the assessment team. Identify an application owner, an operator, and an engineering contact for reproduction and retesting. Confirm preparation effort during scoping.
Where will our code, credentials, and evidence go? Review the Trust Center and on-premises scanning options with your security and procurement teams. Confirm the chosen deployment, model provider, data flows, retention, and access controls before supplying sensitive material; do not assume that every deployment has the same arrangements.
Will the report meet an external assurance requirement? Review a sample report with the party requesting the assessment. Confirm required scope, methodology, tester qualifications or independence, human review, reporting, and retest evidence before changing an existing engagement. Ask Ostorlab to identify the human validation and sign-off included in the proposed service. A technical finding alone does not establish acceptance for every assurance requirement.
Compare Ostorlab with your next manual pentest
Ostorlab's reported advantage over manual pentesting is more critical findings, shorter testing time, and lower assessment cost. The cases show the technical work the platform can perform: investigating credentials across APIs, chaining identity and execution weaknesses, and validating impact with recorded tests. A comparison on your application should establish whether those capabilities improve the results and economics of your current pentest.
Bring one representative application, its connected APIs, and your existing manual pentest scope. Use the scoping discussion to agree:
- Inputs: the build or URLs, test accounts and roles, source access if included, documentation, permissions, and exclusions.
- Deliverables: a report with reproduced findings and evidence limits, access to the investigation record, proposed remediation, and an agreed cleanup and retest handoff.
- Success criteria: validated findings and overlap, usable delivery time, total cost, and engineering effort, using the same definitions for both providers.
- Commitment: a written scope, operating plan, delivery dates, itemized quote, and named owners before testing starts.
Book an Ostorlab vs. manual pentest PoC to agree an equivalent scope, success criteria, and commercial terms for your application. Start with the web sample report or mobile sample report if you want to inspect the evidence format first.