Current limitations

The foundation, BANKING77 data contract, baseline, transformer training path, comparative evaluation, the serving boundary with its real-artifact demo, the CPU validation and acceptance gate, and the delivery documentation are all implemented, and the strict-MVP verdict is now PASS. See the strict-MVP verdict for the audit that issued it. A passing verdict does not make anything below less true: every limitation here is a property of what was measured, not an outstanding task, and none of them is retired by the gate passing.

• BANKING77 is pinned to 1fb62b1bb4635df59a8e1b2f2bc5e0643b2856c8; data preparation validates its source split contract and records local provenance.
• The DistilBERT base model is pinned to 12040accade4e8a0f71eabdb258fecc2e7e948be and one CPU fine-tune has been run.
• The fine-tuned transformer loses to the lexical baseline. On the test split its macro-F1 is 0.6620 against the baseline's 0.8654, a shortfall of 0.2034. This is a measured outcome reported as AC-004 requires, not a defect and not a pending fix. It reflects the frozen configuration in configs/default.toml — two CPU epochs at learning rate 2e-5 — and should not be read as a statement about DistilBERT's ceiling on BANKING77.
• Nothing was tuned in response to that result. Changing a configuration, seed, epoch count, or threshold after observing test metrics would breach the AC-005 leakage controls, so the number stands as measured.
• The curated unsupported-request fixture is a behavioral check, not an OOD benchmark. Twelve hand-written requests across six categories — other-domain, prompt injection, nonsense, empty, non-English, and adjacent-banking — cannot estimate out-of-distribution performance, and an abstention rate over a fixture its author chose is partly a statement about that author's imagination. All 12 abstained, which is a passed check and not evidence of general unsupported-query detection. It is reported only beside the in-distribution contrast (mean confidence 0.0503 against 0.2258 on test, with 0.3201 of test also abstaining), because a total rate on its own is indistinguishable from a model that abstains on everything. No accuracy is reported: no BANKING77 intent is correct for any row, so accuracy is undefined rather than merely unmeasured.
• Neither model's confidence is a probability of correctness. Both are substantially underconfident: the baseline's mean confidence is 0.3770 against an accuracy of 0.8653 (ECE 0.4883), and the transformer's is 0.2258 against 0.6955 (ECE 0.4697). No recalibration was applied; it would require its own validation-only evidence and is out of scope. Abstention is a ranking decision over confidences, which underconfidence does not invalidate — the cost it imposes is coverage, because answers the model would have got right are discarded.
• The transformer's confidence never exceeded 0.6000 on any test example, leaving six of the fifteen bins empty. The persisted threshold of 0.1684 must therefore be read relative to that range and not as an absolute probability.
• Each model's ECE happens to equal its aggregate confidence-accuracy gap here because every occupied bin errs in the same direction. That is a property of this data, not of the metric; the imbalanced fixture separates the two quantities as 1/12 against 1/20.
• Reported ECE is only comparable against another ECE computed under the same binning — 15 equal-width bins, left-closed and right-open with the final bin closed, empty bins excluded from the average.
• The demo is local-only; the acceptance audit is evidenced by a fully evidenced local run and a degraded CI run — by declaration, not as a deficiency awaiting a fix. make demo loads a 257 MB sealed bundle that is deliberately untracked, so a clean runner has nothing to serve. Training one there would produce a different threshold and therefore a different artifact, making the evidence a statement about the runner rather than about the sealed bundle, and it would not fit the 20-minute CPU timeout. The acceptance audit runs in CI in a degraded mode for the same reason: it exercises the ownership contract and the verdict rules, reports each artifact-backed row as not_evidenced with a reason, and separates those environmental gaps from verdict causes, so the CI verdict matches a fully-evidenced local run. AC-013 is therefore Measured locally and not_evidenced in CI. Runs 31104632401 and 31117901793 attempt 2 have executed the acceptance step on a clean CPU runner, so that step's reproducibility is Measured; only the demo remains local-only.
• The CI acceptance step is now enforced, and a step conclusion is still not a substitute for its logs. The step carried continue-on-error: true while the audit's truthful verdict was FAIL. That masking was removed once the verdict became PASS in a fully-evidenced local run, a degraded local run, and a clean CPU runner, each exiting 0: keeping it would have hidden a regression behind a green conclusion, which is the failure mode the gate exists to prevent. A failing audit now fails the job. When reporting a failure, quote the layer you actually ran — a failing recipe ends in make: *** [Makefile:38: acceptance] Error 1 followed by ##[error]Process completed with exit code 2, because make exits 2 when a recipe fails while scripts/validate_acceptance.py itself exits 1. Read the step logs rather than the conclusion, which enforcement improves but does not make sufficient: a conclusion can misreport its step in either direction. Under the masking, run 31104632401 on 4cfc8ec and run 31109003496 on 17c56d9 each printed verdict FAIL with its causes and 14 environmental gaps while every step and the workflow reported success. In the other direction, attempt 1 of run 31117901793 reported failure having executed nothing at all — no runner acquired, zero steps, zero billable runtime, and no lint, test, or audit output. That is an infrastructure non-result, not evidence about the tree. Attempt 2 of that run on d31c333 is the measurement: it audited a tree where U08 is Implemented, printed Strict MVP: PASS with 23 passed, 21 not evidenced, 0 blocked, and its logs contain none of the failure markers above. The PASS in IMPLEMENTATION_STATUS.md is CI-corroborated because those logs were read, not because a badge was green.
• The audit refuses to overwrite a fully-evidenced reports/acceptance.json with a degraded one. Both runs write to the same path, and the report cannot be written beside its evidence without mutating a sibling worktree's preserved reports, so the later degraded run would otherwise silently become the record.
• Measured metrics are the baseline's and transformer's test accuracy, macro-F1, weighted-F1, and per-class figures; the transformer's validation metrics and selected threshold; both models' test coverage, accepted accuracy, and selective risk at that threshold, plus their full risk/coverage curves; both models' expected calibration error and per-bin occupancy; the transformer's accept/abstain decision on each of the 12 curated unsupported requests; and both models' single-request CPU latency. No resource-efficiency metric beyond latency has been measured.
• Latency is descriptive, not a service level. Measured on one machine (CPU, 14 threads, the platform recorded in the report) at batch size 1 over 200 real test texts after 20 discarded warm-up requests: the baseline's p50 sits near 0.55 ms and the transformer's between roughly 8.6 ms and 10.7 ms, with p95 near 11 ms. Ranges rather than single figures are deliberate: latency is the one section of the evaluation report that does not reproduce between runs of the same configuration — the run ID covers the sampling protocol, never the durations — so a precise number quoted here would be stale after the next make evaluate while the metrics beside it stayed byte-identical. That spread is run-to-run variance, not a change in the model. These do not generalise to other hardware. Quote figures from a report you have in hand, not from this sentence.
• Test macro-F1 and weighted-F1 agree to reported precision for both models because the test split is exactly balanced at 40 examples per class, so the weights are uniform. They are not bit-identical — the transformer's values differ in the final floating-point digit because the two averages sum in different orders — but no meaningful comparison distinguishes them on this split. That agreement is a property of the split, not evidence that the two averaging modes are interchangeable; the deliberately imbalanced fixture in tests/fixtures/metric_regression.json is what distinguishes them, and a swap between them fails there.
• Both models were evaluated at the same threshold, which was selected from the transformer's validation confidences. It is not a baseline-optimal operating point, so the baseline's selective figures describe its behaviour at a borrowed threshold rather than its best achievable trade-off.
• CUDA and GPU compatibility are unverified. CI is intentionally CPU-only.
• A synthetic dataset, frozen embeddings, deterministic API predictor, or one-epoch training path would be degraded evidence and cannot satisfy a violated MUST requirement.
• Docker and all deployment work are POST-WEEKEND.

See the authoritative scope controls and the local parking lot for future boundaries.