Measured outcomes

The headline

The validated-repeat study records lower provider charges with 20/24 passing, versus 19/24 for fresh generation and 17/24 for a plain file cache. Cold acquisition costs more than fresh generation. These are small-task lab observations; customer net savings and human productivity remain unproved. The earlier advisory study and older negative/inconclusive pilots remain below.

Guard interception passed a separate staged pilot: exact repeats and same-path variants were denied while different paths were allowed, with $0 model-provider spend. That demonstrates the bounded guard mechanism. Customer incident prevention and invoice reduction require their own evidence.

Validated repeats: lower provider cost in this study

The later 20 October run tests two models, four unchanged toy coding contracts, three conditions and three occurrences: 72 workflows, 24 matched triples and 135 charged generations. Every condition pays for its own acquisition. Answers are never shared between conditions or models. All failing and truncated replies count.

Signed fresh-generation minus AgentEvo is +$0.008448301, or 35.5% lower provider cost in this dataset. File-cache minus AgentEvo is +$0.006243808, or 28.9% lower. AgentEvo passed 20/24 visible rubrics, versus 19/24 and 17/24. Its 13 repeat implementations passed fresh current-contract checks with zero model calls.

Cold acquisition lost to fresh generation: AgentEvo cost $0.009832452 with 6/8 passes; fresh generation cost $0.008992843 with 7/8. On later repeats AgentEvo cost $0.005503848 and passed 14/16, versus $0.014791758 and 12/16. Independent cold sampling, provider routing and the small visible-rubric task set limit causal interpretation. The observed margin over file retention is not a proven general advantage over keeping working files.

Customer net money and human time saved remain unestablished. Setup/review/maintenance labor, local compute price and fees were not measured. Passing reuse checks took a median 0.483 seconds, versus 0.427 seconds for the plain file cache. Automated component totals include learning and setup; lab pacing and receipt waits are not developer productivity.

Qwen3 Coder 30B and GLM 5.3 Flash use the registered temperature 0, 2,000-token requested output cap, five-attempt bound and the same full visible pytest rubric. First submitted payloads match within every model/task cell. Some GLM providers spend the output allowance on reasoning and return no code; those charged replies stay included. Historical ISO-date/retry seeds are disjoint from the target contracts.

The trusted QA adapter exercises actual SDK capture, compilation, verification and reuse with labelled synthetic carriers and isolated Linux validation. This is not a real native-agent journey, hidden-test generalization study or 400-browser capacity test. The T1 prompt has a weekday/date example inconsistency; its explicit weekday-agreement rule and tests were retained unchanged. SDK 0.2.9 ships the tested reuse implementation, but the host must integrate current validation before skipping generation.

All 135 invoices exactly reconcile to $0.060701009 in new key usage. Earlier runs remain counted: $0.079191916 of the original $5 used; $4.920808084 remaining. An earlier Mistral-only interruption completed 18/108 workflows and 40 invoices ($0.005786840); its comparative results remain unknown. The available-provider two-model protocol was registered before inference and does not complete that interrupted three-model experiment. The earlier negative advisory comparison is preserved below.

Earlier advisory comparison: no savings demonstrated

Earlier on 20 October, the advisory-context harness completed all 24 arms / 12 pairs using Mistral Small 24B, Qwen3 Coder 30B and GLM 5.3 Flash through OpenRouter. Their listed input prices were below $1 per million tokens. Both conditions receive the same full visible rubric, temperature 0 and a 2,000-output-token cap, with at most five attempts. The starting condition alternates by task; all failed and non-code attempts count.

Baseline: $0.005498196 and 10/12 rubrics passed. Enabled: $0.006310669 and 9/12 passed. Each uses 26 generations. Signed baseline minus enabled is −$0.000812473: enabled costs about 14.8% more in this dataset. This is a small toy-task observation, not a customer bill or causal estimate.

T1 parses email dates, T2 implements deadline-bound retry, T3 encodes/decodes runs and T4 checks brackets. Fresh disjoint ISO-date and attempt-limited retry seeds produce one actual 1,254-character advisory for each related T1/T2 cell. The summaries omit implementation bodies; novel T3/T4 cells receive the exact recall-miss marker. The seeds pass 8/6 tests. CLI verification reports inconclusive command replay, so its Verified label is not independent correctness or host enforcement.

The six related pairs total +$0.000300795, dominated by Mistral T1 changing from failure after five attempts to success after three. The other five related cells keep the same iterations/pass result and cost more when enabled. The six novel/miss pairs total −$0.001113268; their variation cannot be attributed to recalled algorithms. One observation per cell and provider/cache/order effects prohibit general conclusions.

All 52 actual receipts reconcile to $0.011808865 in new key usage. One generation-less rate-limit rejection recovered with the same payload after bounded backoff; its individual charge remains unknown, while aggregate usage matches. Earlier interrupted attempts add $0.000895202, for $0.012704067 under the same original $5 cap. The later 0.2.8 package release changes documentation/metadata and keeps the SDK runtime payload unchanged; it does not imply a savings fix.

Preparation, review, maintenance, compute and software fees are unmeasured. Elapsed harness time includes receipt waiting and container startup. Customer net savings and human productivity remain unproved. The earlier pilots below have their own methods and budgets.

Token-spending study: negative

Pilot 1 (single model, $0.0256 spend) was inconclusive: signed −$0.0056 with 5 of 8 arms failing — micro-task capability variance swamped any memory signal. Pilot 2, a calibration run ($0.0171 methodology spend), aborted per its pre-registered rule when only one of six candidate tasks qualified as measurable. Pilot 3 ran the longitudinal loop cleanly (three Verified skills accumulated) but recall hit only 1 of 3 eligible occurrences, with no effect in that cell (−$0.000086); the pooled +$0.002981 is almost entirely miss-cell luck.

The cross-model extension (two further models × four rounds, fresh workspaces each) produced three recall-hit comparisons across the models: pooled difference −$0.000137, with unchanged iterations and pass/fail in those comparisons. Total provider spend ≈$0.064 of a $10 cap. A retrieval limit in these tasks was lexical matching. This is a bounded lab finding, not proof of a general bottleneck or a customer savings claim.

Guard interception: proven

The guard pilot ($0 spend) ran six staged failure sessions through the real loop: fixed failures became reusable skills; abandoned tool failures became failure signatures plus regression tests; one abandoned session with code changes additionally compiled a guard. Verified replay of that guard: exact repeat denied, same-path-different-text denied (fail-closed path pin), different path allowed. On real pilot data the report showed 1 catch / 2 estimated avoided calls / $0.000032 estimated value at the session's measured rate. The replay demonstrates a blocked action. The avoided-call valuation is an estimate, not a measured reduction in a provider invoice or human effort.

Loop integrity

The full loop ran end to end against live production: capture, compile, verify, promote, push, cloud recall hit, revoke, tombstone — with the cloud payload sanitized and the end state confirmed. Verifier disagreement correctly blocked one promotion, and the cloud correctly refuses unknown workspaces. The pilots also caught a real product bug: one fail-all session was misclassified as a success and nearly compiled a promotable skill. It was fixed with fail-closed parsing plus regression tests — evidence the loop improves the product, not just stores its history.

Engineering bar

The 0.2.4 train shipped with the full suite green (12,192 passed), lint, format, and type gates clean, deterministic two-pass builds with hash-sealed artifacts verified against the registries, and a post-release smoke suite at 88 passed, 0 failed — including the authenticated cloud path and cold-install journeys from the published packages.

Those are historical release results. The 18 October candidate evidence records newer local checks and remaining production gates separately.

Run your own pilot

The local memory and guard paths need no Agentevo account or sync upload. The matched model-spending pilots send task prompts and context to your chosen model provider using your own key and incur its charges. Use suitable non-sensitive tasks, run the savings, guard and longitudinal comparisons, and include setup, review and failed attempts. These figures describe our lab; customer savings and production capacity need their own evidence. If your numbers disagree with ours, we want to know: support@agentevo.sh.