Engineering evidence · dated summary
What we checked. What remains open.
Local software checks and staged examples support specific behavior. They do not establish customer net savings or general production capacity. This is a curated summary of internal QA, not a publication of private logs or an independent audit.
Regression and installation
Before packaging SDK 0.2.9, the source regression closure accounts for 12,803 passes and 32 skips, with no unresolved errors and 100% measured coverage of 36,807 statements and 12,548 branches. This combines the complete run with focused fixture-cleanup repairs; it is not a claim of one error-free full-suite invocation. All 577 pinned source/test inputs matched at the comparison's completion.
The installed SDK 0.2.9 gate passes 1,657 cases, one skip, zero failures. The 16-case deployment-backup module is outside the owner's requested scope; the retained unfiltered run had a backup-fixture failure and two premature-public-version recipe failures. Public pins were corrected and the full in-scope installed suite reran. The remaining skip needs Linux PID namespaces unavailable on macOS. All 43 shipped SDK runtime files match the source used by the reuse study.
Public PyPI and npm 0.2.9 downloads match their selected hashes. Each fresh installation passes seven version/init/repeat/doctor/offline-recall/read-help/local-schema checks with isolated state and a retained synthetic key. The actual interpreters are Python 3.12.13 and 3.14.6 respectively; a separate 3.10.20 runtime/import/schema smoke passes. Real OS keychain, host connection and protected Clerk ceremonies remain separate acceptance. Use the one-command local start.
Prior SDK 0.2.8 release checks
The 20 October source gate completed 12,731 tests passed, 32 skipped, zero failures. The explicit 100% statement/branch gate covered 36,630 statements and 12,488 branches. All 632 frozen inputs matched at completion. Skips are not passes, and coverage does not prove absence of defects.
Separate SDK 0.2.8 installed checks passed 1,601 cases with one recorded skip. Its 41 runtime payload files match the preceding 0.2.6 candidate; the new release updates documentation and version metadata. The frontend suite passed 73 cases using synthetic identity/API fixtures. These boundaries overlap and are not added into a larger total.
Fresh public PyPI and npm 0.2.8 installations each passed version, first initialization, repeat initialization, doctor and empty local recall on macOS arm64/Python 3.12. They use isolated state and one retained synthetic vault key. Real OS keychain, host connection and other platforms require their own acceptance. Use the installation guide.
Product acceptance
The repository's 32 automated spec-acceptance cases cover the original product, data-flow and API security contract. Wider current checks exercise the learning lifecycle, scoped authentication, sync, jobs, quota, export/deletion and revocation with controlled fixtures.
For a recorded synthetic failure, the published CLI created a failure signature, regression test and guard. Repeating the edit required approval; the test action was allowed. This demonstrates the bounded guard mechanism, not automatic host enforcement or an avoided customer incident. Real-account and physical second-machine journeys remain separate.
Tenant and workspace isolation
130 targeted synthetic CRUD/upsert canary cases exercise all 26 tenant-scoped application tables, alongside API identifier, export and delivery-authority checks. Foreign-resource and scope refusals are exercised with controlled principals. SQLite uses application guards rather than native row-level security.
These tests do not authorize direct database operators or prove a real provider's organization/role configuration. Actual Clerk membership, role changes and revocation need their own staging acceptance.
Data and trust boundary
There is no raw-transcript sync route in the hosted API. Local capture remains local by default. Searchable sync is opt-in, minimized, tenant-bound and signed. Opaque Relay uses workspace encryption. Explicit cloud recall can send query/context, and optional BYOK embeddings contact the chosen provider.
Secret detection is bounded and can miss sensitive material. Review before upload. Retrieved content is advisory; signatures identify a source, verification supplies bounded evidence, and promotion does not independently prove task correctness. Offline revocation propagation can lag until receipt/sync.
See the Privacy Policy, Terms and reviewed host integration guide. Hosted key custody and operator/privacy facts require their own operating acceptance.
A 400-session native Linux rehearsal
Two isolated native x86 Linux replays each completed 3,200 application requests, including 800 correctly owned writes, with zero errors or retries. Each replay made 9,600 real uncached Clerk session GETs. The fixture starts with 400 Free tenants and 12,000 objects and covers usage, analytics, recall and signed sync over TLS with encrypted storage.
| Replay | Requests completed | p95 latency | Maximum | Errors / retries |
|---|---|---|---|---|
| First | 3,200 / 3,200 | 29.067 s | 30.531 s | 0 / 0 |
| Repeat | 3,200 / 3,200 | 30.187 s | 31.258 s | 0 / 0 |
The tested software policy is 32 active requests, 1,024 pending, a 30-second queue wait and a 40-second total client deadline. Fixture tickets and 600-second tokens are explicit setup. These are controlled sessions, not 400 rendered browsers. Normal token refresh, browser fan-out, sustained traffic, shared provider quota and an accepted operating SLO remain separate gates.
Earlier 1,000-user synthetic profile and its retained timeouts
Earlier synthetic PAT workload
The final local fixture used 1,000 synthetic PAT users, 100 tenants, 3,000 initial objects and 10,000 requests per profile. The mix was 20% usage, 60% recall and 20% signed sync, against one local Uvicorn process and SQLite WAL. No other test job ran during measurement.
| Concurrent requests | p95 latency | Errors / 10,000 |
|---|---|---|
| 50 | 1.151 s | 0 |
| 100 | 1.481 s | 0 |
| 250 | 2.492 s | 0 |
| 500 | 5.218 s | 2 read timeouts |
39,998 of 40,000 requests succeeded. Two timeouts at the unchanged ten-second phase deadline remain an unresolved stress boundary. Real Clerk, TLS/proxy, MCP streams and background loops were outside this capacity fixture. This is 1,000 registered synthetic users, not 1,000 simultaneous requests or a hosted performance promise. Client idle-expiry changed between older and final runs, so their tables are not a pure product-optimization comparison.
Validated repeats: a conditional provider-cost saving
The completed two-model study covers four toy contracts, three conditions and three occurrences: 72 workflows and 135 billed generations. Each condition pays for its own acquisition. All failures and truncated replies count.
| Condition | Provider USD | Passed | Model calls | Validated reuses |
|---|---|---|---|---|
| Fresh generation | $0.023784601 | 19/24 | 58 | 0 |
| Plain file cache | $0.021580108 | 17/24 | 46 | 11 |
| AgentEvo reuse | $0.015336300 | 20/24 | 31 | 13 |
AgentEvo's provider charges are 35.5% lower than fresh generation and 28.9% lower than the file cache in this dataset. Its cold phase costs more and passes 6/8 versus 7/8; later validated repeats recover that expense. All charges exactly match key usage: $0.079191916 total lifetime spend under the original $5 limit, including earlier interrupted work.
Customer net ROI, human time saved and a general advantage over retained files remain unestablished. The trusted QA adapter uses actual SDK behavior and labelled synthetic history; this is not a native-agent customer journey or a 400-browser load result. Read every matched triple, cold losses and method limits; see the optional host integration requirements.
Earlier advisory comparison: no savings demonstrated
The earlier 20 October advisory comparison completed three models × four toy coding tasks × two conditions: 24 arms, 12 pairs and 52 charged generations. Both conditions receive the same full visible test rubric. Costs include failing, truncated and non-code replies.
| Condition | Provider charges | Visible rubrics passed | Attempts |
|---|---|---|---|
| Baseline | $0.005498196 | 10 / 12 | 26 |
| AgentEvo enabled | $0.006310669 | 9 / 12 | 26 |
Enabled costs $0.000812473 more, about 14.8%, in this small run. The generation ledger exactly matches the key-usage delta. Earlier interrupted attempts remain included in the $0.012704067 total expense under the original $5 cap. One observation per cell, sparse advisory text, order/provider/cache variation and differing success outcomes prevent a general causal claim.
Customer setup, review, maintenance, compute and fees are unmeasured. Whole-condition time and net ROI remain unknown. Read all 12 task pairs and the method limits.
Release and value limits
The selected packages are published and their public bytes are verified. Production rollout is assessed separately. Hosted readiness retains protected sign-up/reset and identity-policy journeys, shared provider/workload acceptance, routed alerts, operator/privacy facts and scoped rollback evidence. Passing local tests does not close those rows.
Customer money and human time savings are unproved. The completed repeat-work comparison observes lower provider charges with current validation, but cold acquisition loses and customer costs are unmeasured. The earlier advisory comparison reports higher enabled charges and fewer passes. Recall-miss variation cannot be attributed to retrieved algorithms. Read the full bounded outcome summary.
Questions about method or a reproducible result? Contact us about the evidence. Do not send credentials or private transcripts.