IntentGuard in twelve questions

Confidence-aware support intent classification on BANKING77: one lexical baseline, one fine-tuned DistilBERT, reproducible evaluation on an untouched test split, abstention driven by a validation-selected threshold, and a typed FastAPI boundary that serves one sealed artifact. It is a compact portfolio project built to show an honest ML lifecycle end to end — explicitly not a production service. Project overview

Support automation should not act on requests it does not understand. The user stories behind the system ask for validated data (US-001), a meaningful baseline comparison (US-002), safe abstention so uncertain requests route to humans (US-003), typed local inference (US-004), and verifiable claims (US-005). Requirements § User stories

Five stages, in execution order: pinned data (BANKING77 at a fixed revision, canonical splits, recorded provenance) → two models → a validation-only threshold sealed into an immutable artifact → evaluation on the untouched test split → serving that loads the same sealed artifact. The threshold is the spine: selected once from validation data, then loaded unchanged by evaluation and serving alike. README § lifecycle · ML system design

“Improvement” needs an explicit reference point. ADR-0001 selects DistilBERT for real transformer fine-tuning within one weekend on local hardware, and the TF-IDF baseline keeps the comparison honest. The measured outcome: the baseline wins by 0.2034 macro-F1, reported as measured rather than retuned away. ADR-0001 · Implementation status

When maximum predicted probability falls below the persisted threshold (0.16841767053420467), the API returns decision="abstain" with intent=null instead of guessing. Confidence here is a ranking signal for abstention, not a probability of correctness — both models are substantially underconfident, and the repository says so explicitly. FR-004 · README § confidence

AC-005 requires threshold selection to use validation predictions only and never read test labels. After the test result was observed, no configuration, seed, epoch count, or threshold was changed — the repository treats retuning-after-seeing-test as a breach of its own controls, and the negative result stands. AC-005 · Limitations

make evaluate loads both sealed bundles, applies the persisted threshold without reselecting it, and writes machine-readable evidence: comparable metrics, selective-prediction figures, calibration over fixed bins, risk/coverage curves, a curated unsupported-request check, and single-request CPU latency. Reports are generated and untracked — reproduce them rather than trusting copies. Implementation status § U05

make serve exposes typed /health and /v1/predict endpoints over one sealed transformer bundle, verified by checksum before the port is bound. Serving has no code path that trains or rewrites an artifact. The demo's abstained confidence agrees with the evaluation's value for the same fixture row to seven decimal places — which is what shows serving and evaluation read the same weights and threshold. Interface contract · Operations

make acceptance audits all 42 primary identifiers (tasks, FRs, NFRs, ACs) and prints an enumerated verdict; it exits zero only when no cause remains. The current strict-MVP verdict is PASS — issued only after the U08 delivery evidence existed, because editing the status first would have produced a passing verdict by declaration. The verdict · Claim Ledger

The governed agent workflow the repository was built with: six command contracts (prime, brainstorm, plan, implement, validate, handoff), each with a declared mutation level, shared safety rules, and executed-evidence requirements. FPAT review path · FPAT overview

AGENTS.md defines required startup reading, a single authoritative specification with declared precedence, a status vocabulary where Measured never substitutes for implementation, and approval rules that keep remote mutations behind explicit, current consent. Agents review path · AGENTS.md

The repository's own artifacts — not this portal — carry the evidence: a pinned data contract, leakage-controlled threshold selection, a negative result reported prominently, an acceptance audit that measures itself honestly, and a governed agent workflow. The Claim Ledger maps every claim to its support; the evidence-chain graph makes the requirement→proof paths navigable.