Implementation status

Status uses the repository vocabulary in AGENTS.md. Measured is an evidence qualifier and never substitutes for implementation.

Strict MVP is complete: every MUST capability U01–U08 is Implemented and no applicable claim is left unmeasured. The pinned dataset contract, both models' test metrics, the transformer's validation metrics and selected threshold, the test-split comparison and abstention behaviour, the curated unsupported-request check, both models' calibration and single-request CPU latency, and the served accept/abstain behaviour of the sealed transformer artifact over HTTP are Measured from executed local runs. No GPU claim is evidenced.

The five-minute walkthrough is Measured#

Executed on this tree against the E05 artifact root, following README.md's "Review in five minutes" section in order. Elapsed command time, measured with a wall clock:

Total command time 38 s, well inside the five-minute boundary, which excludes prerequisite installation. make setup is quoted against a warm environment: a first-time clone additionally pays the locked-environment install, measured separately at roughly 25 s on this machine including interpreter provisioning. Stage 5's non-zero exit is recorded rather than smoothed over — the walkthrough deliberately preceded U08's promotion, so the gate was still reporting the truth about an unfinished capability. The re-audit after promotion is below.

The demo's observed output, against bundle intentguard-distilbert-1fb62b1bb463-88e538757339: /health returned ready on device cpu with label_count 77; "How do I activate my new card?" was accepted as activate_my_card at confidence 0.5300218231428692; the curated row unsupported-001 abstained at 0.04127836201977225 with intent null. Both used the persisted threshold 0.16841767053420467. That abstained confidence agrees with the E05 evaluation's 0.04127837051435484 for the same row to seven decimal places, which is what shows serving and evaluation read the same weights and the same threshold — a matching model_version string alone would not.

U06 is Implemented, which closes the serving half of the MUST hierarchy: the typed FastAPI boundary loads one sealed artifact, applies the persisted validation threshold, and make demo demonstrates one accept and one abstain over a real socket rather than through a test client or a deterministic stand-in.

Suite counts reconcile across three environments#

An earlier revision of this document recorded make test as 585 passed with the artifact present and 563 passed / 22 skipped without it. Those figures do not reproduce on any tree and are corrected here. The collected total is what changed: this row's counts were carried forward from the tree that measured them and were never re-measured as tests were added.

Each row names the tree it was measured on, because a count carried across trees is exactly how the 585/563 figures went stale. The current tree collects 650, adding 13 audit tests to the 637 collected at 4cfc8ec:

Within one tree the numbers differ only by which gate is unmet, and each gate's skip message names the command that satisfies it. The CI row was once an arithmetic expectation of 22 + 17 = 39 skips; run 31104632401 measured exactly that against 4cfc8ec, so it is a measurement and must not be re-labelled an expectation.

The 17c56d9 row was likewise recorded as an expectation first: the two gates are unchanged, so 650 collected minus 39 skips predicted 611 passed. Run 31109003496 measured exactly 611 passed, 39 skipped, 1 warning in 21.10s, so it is now a measurement.

The d31c333 row is a measurement from the outset: run 31117901793 attempt 2 measured 611 passed, 39 skipped, 1 warning in 19.14s. It matches 17c56d9's counts because E08 changed no test count — the two documentation-only commits between those trees added no test and removed none, and the one assertion narrowed in tests/unit/test_validate_acceptance.py left the collected total at 650. Every CI row is pinned to its tree, because a count carried across trees is exactly how the 585/563 figures went stale.

Strict-MVP verdict: PASS#

Issued by make acceptance, which exits zero only when no cause remains. Measured on this tree after the walkthrough above: the audit classified 44 rows over 42 primary identifiers — NFR-001 contributes three rows, one per clause — and reported 43 passed, 1 not evidenced, 0 blocked, with an empty cause list.

The order matters and is recorded deliberately. The same audit on the same tree reported FAIL — 3 cause(s) before U08 was promoted, and PASS after. The three causes were U08 Planned, T-008 blocked, and NFR-010 blocked — the latter two solely because their owning capability was Planned, since README.md and docs/OPERATIONS.md already existed. That is precisely why the promotion came last. U08's declared status is the audit's only lever over those two rows, so editing it first would have produced a passing verdict by declaration; the walkthrough, the demo against the real sealed artifact, and the 19-figure metric diff are the evidence the promotion rests on.

Causes 2 and 3 were the audit's own correction to an earlier defect in it. Its first version passed any identifier whose named file was present on disk, consulting the owning capability's status only when the file was missing. That reported NFR-010 as passed while U08 was Planned — a Planned capability rendering as satisfied, which is precisely the substitution this gate exists to prevent. The status decides, and presence alone never passes.

The verdict is environment-independent, and CPU CI has now confirmed it#

Three audit runs on this tree exit 0 and report PASS — two local, one on a clean CPU runner:

The degraded run reaches the same verdict while routing 21 artifact-backed rows to environmental gaps, which is the property the audit was built to have. CI's counts are identical to the local degraded run's, row for row, which demonstrates that property in a second environment rather than asserting it.

That CI run is the first clean-runner audit of a tree where U08 is Implemented. Its foundation job acquired a hosted runner, executed all seven steps, and measured make lint clean — Ruff, mypy strict over 52 files, 10/10 foundation checks — make test at 611 passed / 39 skipped in 19.14s, and the acceptance audit at the verdict above. The audit recipe exited 0 on its own merits: the step logs contain no ##[error], no make: *** [Makefile:38: acceptance] Error 1, and no ##[error]Process completed with exit code 2. For the first time in this repository the step's green conclusion and its true exit status agree.

Attempt 1 of that same run is recorded here rather than discarded, because omitting it would overstate the evidence. It concluded failure at the workflow level without ever executing: no runner was acquired, the job was cancelled with zero steps and zero billable runtime, and it produced no lint, test, or audit output. That is an infrastructure non-result — not a failed check, and not evidence about this tree in either direction. It is the mirror image of the masking problem below, and the same lesson: a conclusion is never read in place of logs.

The CI acceptance step is now enforced. It carried continue-on-error: true while the truthful verdict was FAIL — a masked step whose real exit status stayed visible in the logs, rather than a || true that would have discarded it. That masking was removed once the run above measured PASS on a clean runner, because a flag that once prevented a red check from restating what the documents already said would now hide a regression behind a green conclusion. A failing audit fails the job, and .github/workflows/ci.yml also declares workflow_dispatch so an exact ref can be revalidated after an infrastructure non-result without manufacturing a commit. Enforcement does not retire the reading rule: this verdict is trustworthy because the step logs were read, as docs/LIMITATIONS.md records, and not because a badge is green.

Four causes were removed as circular, not as fixed#

An earlier revision recorded seven causes. The other four — U07 Partial, plus T-007, NFR-002, and NFR-009 blocked on that status — were an artifact of the gate measuring itself, and removing them changed no evidence:

• T-007, NFR-002, NFR-009, and AC-014 are E07's own deliverables, so the audit is the thing they name. Deciding them from the owning status closed a loop: a FAIL verdict was the stated reason U07 was Partial, and a Partial U07 then blocked the three rows, so one declared status became four causes and a truthful FAIL described itself four extra times. AC-014 already had this carve-out; it now covers the three identifiers that share the problem. They are classified on observable orchestration — the script exists, the workflow runs make test and make acceptance, the run covered all 42 identifiers — and never on the verdict's value.
• U07 became Implemented on executed evidence, not on a document edit. Run 31104632401 is a measured CPU execution of the whole orchestration. Requiring a PASS verdict to promote U07 was unsatisfiable by construction: a green step needs continue-on-error: true removed, which needs the verdict to already be PASS, which needs U08 Implemented — so U07 could never close on its own work. A capability is complete when its own behaviour exists and has run, which is what the vocabulary in AGENTS.md says and what the CI run shows.

The verdict stayed FAIL throughout that reconciliation. The three causes that remained were all genuinely downstream of U08, and each would still have been reported if the audit's own rows had been removed entirely. U08's delivery, recorded above, is what cleared them.

The reconciliation has since executed on a clean CPU runner. Run 31109003496 on commit 17c56d9 printed Strict MVP: FAIL — 3 cause(s) naming exactly U08, T-008, and NFR-010, with 14 claims separated as unverifiable in that environment and not counted against the verdict. So the three-cause verdict is reproduced by a second, independent environment: the runner has no artifact root, routes 21 rows to not_evidenced where a fully-evidenced local run passes them, and still reaches an identical cause list. That is the environment-independence property the audit is built to have — the causes are substantive, and the environmental gaps are reported separately rather than inflating them. That property holds after U08's delivery too: both environment states now reach PASS, as the verdict section records.

Three further candidate causes were assessed and did not hold:

• NFR-001's RAM clause passes as Measured. The sealed bundle's provenance.json records peak_memory_bytes 2,877,472,768 — 2.88 GB against a 24 GB clause.
• NFR-001's GPU clause is not_applicable, not an unmeasured MUST. The same record shows cuda_available false on device cpu, so no GPU path was exercised. NFR-002 requires CPU execution; the GPU clause describes local convenience hardware, not a deliverable capability. CUDA compatibility is unverified and no GPU claim is made anywhere. This is the single not_evidenced row.
• The 585/563 contradiction is resolved above rather than left standing as a verdict input.

No applicable MUST clause is left unmeasured: across all 44 rows, 25 are measured with an evidence path and 19 are not_applicable with a recorded reason. make lint enforces that no row can be excused without one.

NFR-006's latency figures remain Measured but explicitly non-reproducing: the run ID covers the sampling protocol and never the durations. The audit records that qualifier rather than a bare pass.

The serving evidence above was produced on this machine from the E05 artifact root. It is not a CI claim: the demo has not been executed in GitHub Actions and will not be, by decision D39 — the sealed bundle is untracked and roughly 257 MB, and a CI-trained model would carry a different threshold and so be a different artifact. make demo on a clean CPU runner is therefore out of scope rather than pending, and it is not an unmet part of U07. The latency figures in the demo transcript are single observations and no service-level claim is made from them. Those figures are also narrower than their published definition, which is documented divergence D2: INTERFACE_CONTRACT.md:72 defines latency_ms as request validation plus inference plus response assembly, while the served value times inference only. Validation has already completed when the handler is entered, and assembly cannot be inside a number the response being assembled must carry. The reported value is therefore a lower bound on the contract's quantity, recorded rather than relabelled.

The transformer does not beat the baseline#

The fine-tuned DistilBERT scored 0.2034 lower on test macro-F1 than the TF-IDF baseline. REQUIREMENTS.md AC-004 requires the comparison to be reported whichever way it falls, so this is a satisfied acceptance criterion and not a defect. At the frozen configuration — two CPU epochs, learning rate 2e-5, max sequence length 96 — the lexical baseline is the stronger model on this split.

No configuration, epoch count, seed, or threshold was changed in response to observing this result. Doing so would let a test label influence a modelling decision, which AC-005 forbids. The result stands as measured, and any future attempt to improve the transformer must be justified and validated without reading the test split.

Neither model's confidence is a probability of correctness#

Both models are substantially underconfident on the test split. The baseline reports a mean confidence of 0.3770 against an accuracy of 0.8653, and the transformer 0.2258 against 0.6955. Every occupied bin errs in the same direction for both models, which is why each ECE happens to equal its aggregate confidence-accuracy gap here; that is a property of this data, not of the metric, and the imbalanced fixture in tests/fixtures/metric_regression.json separates the two quantities (1/12 against 1/20).

The transformer's confidence never exceeded 0.6000 on any of the 3,080 test examples — six of the fifteen bins are empty. The persisted threshold of 0.1684 therefore has to be read relative to that range, not as a claimed probability of correctness. Reported because a reader who takes 0.1684 as a probability would conclude the system accepts almost anything, which is not what the coverage of 0.6799 shows.

No recalibration was applied. Temperature scaling or an equivalent fit would need its own validation-only evidence, and it is out of scope for the weekend MVP. Abstention here is a ranking decision over confidences, which underconfidence does not invalidate; the cost it imposes is coverage, since answers the model would have got right are discarded.