What we have measured.

Different outputs for identical detection requests

In this pair, 14 of 14 detection request bodies were byte-identical. Their parsed replies differed in 14 batches, and their finding sets differed in 14. The final model recall decreased despite identical detection inputs. These two runs cannot estimate repeatability or attribute the full change to verifier code. Native coverage and model coverage remain separate above.

Download source-free request and reply hash comparison. This offline comparison made no additional provider calls.

Separate fresh model-evaluation sample

Twenty-six cases were hash-ranked across 13 CWEs, one vulnerable and one safe case each, after excluding all 140 earlier model-evaluated IDs. All five vulnerable SQL-injection cases in the fixed author corpus had already been measured. Both SQL-injection strata were excluded to preserve balance without reusing a case. SQL-injection remains covered by the 112 known controls above.

The evaluator reviewed source anchors before predictions and kept the selected source from implementation agents until candidate freeze. Public-suite labels and the deterministic corpus had already been used during development. This is not an independent customer-repository holdout.

The final result repeats the same 26 cases after the first attempt left 10 verification stages failed and 0 not run. Scanner source, route bytes, labels, sink anchors, context and grouping are unchanged. This rerun is not a new untouched sample. Both attempts, their failures and their full costs remain visible below.

The fresh verified model missed 4 vulnerable examples and flagged 0 safe examples. The union missed 0 and flagged 0. Failed and not-run stages remain unmeasured.

Fresh 26-case receipt.

New authorization and preserved cost history

All 5 new runs made 302 provider requests. Their calculated bound is $0.8771739 under the separately authorized new $1 cap. Earlier evaluations cost $0.9716970, making the lifetime calculated bound $1.8488709. The new cap is separate from the earlier budget; no historical spend is erased or counted twice. Actual account billing was not independently verified.

The series contains 166 distinct planned cases. Repeated baseline/candidate measurements do not count as new cases. Failed and not-run cases remain explicit. Detection and native analysis use isolated routes; verification can inspect bounded complete source dependencies. Other scan lanes, hosted runtime settings, repair behavior and after-merge outcomes remain unmeasured.

Retained paid trial trial-v15, 112 planned cases, at most 4 files per batch across 28 batches, 105 requests, calculated bound $0.3002832. Verified counts: 36 / 20 / 2 / 54; failed 0, not run 0.

Retained paid trial fresh-initial, 26 planned cases, at most 8 files per batch across 4 batches, 24 requests, calculated bound $0.0766476. Verified counts: 6 / 3 / 0 / 7; failed 10, not run 0.

What changed in the evaluation input

Detection prompts and native analysis still receive only the isolated route files. Verification can now inspect bounded, complete directly imported dependency bodies from the actual corpus. No native findings or expected labels are injected into model prompts. This is context-assisted verification, with changed input scope compared with the prior isolated verification result. The new single run also groups the earlier two cohorts together, so batching and stochastic response variation limit causal comparisons.

The frozen input-context manifest contains 5 dependency modules; a review can receive at most two complete modules, 3,000 characters each and 6,000 characters total. Manifest SHA-256: f313c2303435ead2118e2588be660112b1cd72e908afec942dd3e9c8ca56ec57. The raw receipt retains individual source and sanitized-source hashes. This does not measure full repository scans, customer accuracy, model repair correctness or after-merge behavior.

Later focused verification-only replay

The later verifier replay includes 12 deliberately selected existing candidates: four false-positive examples, four prior-miss candidates and four retained vulnerable controls. It made no fresh detection calls or native inferences. These are known candidate decisions, not a new sample or a remeasurement of the 112-case aggregate above.

Changes to candidate decisions cannot recover vulnerabilities absent from the detector output. No updated production detection aggregate is claimed for this later verifier source. The full recorded 112-case result and its cost remain unchanged.

Separate verification-only replay receipt, 12 requests, calculated bound $0.0229683.

Later detector check: four known full-pipeline controls

This later detector revision is checked only on four fixed known cases: safe cases BenchmarkTest00991 and BenchmarkTest00071, and vulnerable cases BenchmarkTest00265 and BenchmarkTest00497. Detection, mapping, source-context verification and native union run together on these four unchanged inputs. This intentionally small check does not remeasure the 112-case aggregate above or establish a production-wide improvement. The recorded full-run regression and the separate 12-candidate verifier replay remain unchanged.

The verified model detected 1 vulnerable controls, still missed 1, and flagged 0 safe controls. The native and model union detected 2 vulnerable controls and flagged 0 safe controls. Failed and not-run outcomes remain separate above.

Focused four-case full-pipeline receipt, 5 requests, calculated bound $0.0137376. Frozen protocol SHA-256: 0fe0459042ca32b635cbed6a635efdfcf4a8af5617f4dcb44e4cc1c4a2c4625c.

Incremental budget and complete history

All 3 new attempts made 96 provider requests. The new calculated cost bound is $0.2744592 of the incremental $0.30 cap. Earlier runs cost $0.6972378, bringing the cumulative bound to $0.9716970 under the $0.9972378 ceiling and original $1 cap. Actual account billing was not independently verified.

The series still includes 140 distinct planned cases, including the earlier 28-case preliminary trial retained in the unchanged parent receipt. Rerunning these 112 cases does not add new cases to that total.

Focused four-case full-pipeline attempt 1, 5 requests, bound $0.0137376. Verified counts: 1 / 1 / 0 / 2, scored 4 of 4, failed 0, not run 0.

Fresh 28-case model evaluation

One vulnerable and one safe example per CWE, hash-ranked after excluding all 112 earlier model cases. An evaluator reviewed operation anchors before predictions without sharing this selected source with prompt or mapping implementers. The public suite labels and deterministic corpus were already used in development. This is not an independent customer repository holdout.

The fresh verified model missed 2 vulnerable examples and flagged 0 safe examples. The served union detected 14 vulnerable examples and flagged 0 safe examples. These counts do not establish precision across all emitted findings or predict accuracy on your repository.

These cohorts contain 140 distinct planned cases. The other 1,090 suite cases were not selected; failed and not-run cases above remain unmeasured. Prior familiarity and model training contamination remain unknown.

Shared cost cap and measured scope

All 4 new runs made 168 provider requests. The new calculated peak cost bound is $0.4532796. Earlier published runs cost $0.2439582, bringing the cumulative bound to $0.6972378 under the shared $1 cap. Returned token usage or conservative reservations account for every declared attempt. Actual account billing was not independently verified.

The 84-case run used 11 batches capped at 8 files; the fresh run used 4 batches capped at 8. Configured model deepseek-v4-flash; provider returned deepseek-flash. Actual production runtime configuration was not independently verified.

28 untouched model-evaluation cases

One safe and one vulnerable example per CWE, excluding every earlier model-evaluated ID. The evaluator fixed the hash-ranked selection and reviewed operation anchors before predictions. The public suite and deterministic corpus were already familiar from prior development. This cohort is separate from the familiar rerun; it is not an independent repository holdout or customer accuracy estimate.

84 distinct cases have now received model predictions from the 1,230-case suite. The remaining 1,146 cases have no model measurement in these cohorts. Model training contamination and prior familiarity remain unknown.

Cost and measured scope

All 4 follow-up runs made 82 provider requests. The final pair cost bound was $0.1056558 and earlier trials cost $0.1083948, for a follow-up total of $0.2140506. 0 requests lacked returned usage. Including the original pilot, the cumulative calculated bound is $0.2439582 under the shared $1 cap. Account billing was not independently verified.

Configured model deepseek-v4-flash; returned models deepseek-flash. Concurrency 1 serializes these cost-controlled runs. The final familiar run used 4 batches and the fresh run used 4, each capped at 8 files. Actual production runtime configuration was not independently verified. Imported helper bodies, application execution, additional scan lanes, patch generation, PR creation and full hosted scan accuracy remain outside this measurement.

Recorded model pilot, October 21

28 selected cases measured, 1,202 not selected.

One vulnerable and one safe example from each of 14 CWE categories, hash-ranked before inference. This small, balanced, public label-known sample cannot estimate customer accuracy. Labels and suite identifiers were removed from model messages. All four lanes below use the same selected cases and exact expected CWE plus reviewed sink lines; unrelated findings receive no credit.

The model alone missed 8 of 14 vulnerable examples and flagged 2 of 14 safe examples before verification. Verification removed two false positives and one true positive in this pilot. The served union preserved native findings and added no expected-case detection credit. A passing combined result here does not establish that the model improves coverage.

Code-default model alias deepseek-v4-flash; provider returned deepseek-flash. Explicit concurrency 1 serializes this cost-controlled pilot; application default is 24. Production runtime configuration was not independently verified. Imported helper bodies are absent here, so this pilot is separate from the full-suite context-aware deterministic run.

Provider requests

15

Measured input / completion tokens

34,524 / 16,292

Calculated peak cost bound

$0.0299076, $1 run cap

Unknown-usage requests

0

Summed request latency

81.38 seconds, local pilot

Cost uses frozen provider price bounds, checked 2026-10-22T02:34:26.434840+00:00. Completion tokens include thinking tokens. Account billing and full hosted scan costs were not independently verified.

Earlier OWASP Python development regression

On the same 1,230 known cases, this run detected 452 labelled vulnerable cases, missed 0 and flagged 40 labelled safe cases. 0 unsupported and 0 failed cases remain explicit. The previous recorded run had 452 detections, 0 misses and 40 false positives.

Single-file deterministic matching, models and verification disabled. These labels informed development. This does not measure repository-level accuracy or OWASP endorsement.

Generated repair consumer regressions

15 of 15 authored checks passed, 0 failed. Two deterministic patches were generated for JavaScript and Python consumers. Configured output was preserved; missing configuration failed closed. Static-data, cursor, browser, unknown-runtime and missing-import inputs abstained. A saved placeholder was blocked, while an unchanged cursor passed publication review only with original source available.

101 ms locally, $0 model cost. These synthetic consumers measure a narrow repair policy. They do not measure exploit closure, provider rotation, external application regressions or model-generated patch correctness.