Review path · ML system
The model lifecycle, stage by stage
Every stage links to the specification that defines it and the measured evidence that proves it ran. Latency and calibration caveats are part of the story, not footnotes.
Data — a pinned contract, not a download
BANKING77 is pinned to revision 1fb62b1…. make data validates the source split contract (10,003/3,080), derives validation from training data only (8,502/1,501/3,080 across 77 labels), computes deterministic fingerprints, and records local provenance. FR-001 · U02 evidence
Baseline — TF-IDF + logistic regression
The baseline is measured from its reloaded artifact, not the in-memory model: test accuracy 0.8653, macro-F1 0.8654, seed 42. It exists so that “improvement” has a reference point — and on this split, it is the stronger model. FR-002
Transformer — one fine-tuned DistilBERT
Threshold — selected from validation only, then sealed
make train selects the abstention threshold (0.16841767053420467) from validation confidences only, at coverage 0.7015, and seals it inside the immutable artifact. Evaluation and serving load it unchanged; test labels never influence it. AC-005
Test evaluation — the untouched split decides
Both sealed bundles are compared on the same test IDs, at the same persisted threshold. The comparison verdict recorded in the report is baseline_better:
| Metric | TF-IDF baseline | DistilBERT |
|---|---|---|
| Accuracy | 0.8653 | 0.6955 |
| Macro-F1 | 0.8654 | 0.6620 |
| Coverage at threshold | 0.7500 | 0.6799 |
| Accepted accuracy | 0.9333 | 0.8161 |
| Selective risk | 0.0667 | 0.1839 |
| Expected calibration error | 0.4883 | 0.4697 |
Selective prediction — what abstention buys and costs
At the persisted threshold the transformer covers 0.6799 of test with accepted accuracy 0.8161 (selective risk 0.1839); the baseline covers 0.7500 at accepted accuracy 0.9333. Both models are underconfident, so the cost of abstention is coverage: answers the model would have got right are discarded. Full risk/coverage curves are written for both models.
Calibration — reported, not repaired
ECE 0.4883 (baseline) and 0.4697 (transformer) over 15 fixed equal-width bins. No recalibration was applied — temperature scaling would need its own validation-only evidence and is explicitly out of the weekend scope. The transformer's confidence never exceeded 0.6000 on any test example, so the threshold must be read relative to that range. Limitations
The unsupported-request check — behavioral, not OOD
Twelve hand-written requests across six categories all abstained at the persisted threshold. The repository refuses to call this OOD detection: it is reported only beside its in-distribution contrast (mean confidence 0.0503 on the fixture vs 0.2258 on test, where 0.3201 of test also abstains), because a total rate alone is indistinguishable from a model that abstains on everything. AC-012
Latency — descriptive, never a service level
Single-request CPU latency at batch size 1 over 200 real test texts after 20 warm-up requests: baseline p50 near 0.55 ms; transformer between roughly 8.6 and 10.7 ms across runs. It is the one section of the evaluation report that does not reproduce between runs — the run ID covers the sampling protocol, never the durations — so the repository quotes ranges and tells you to read your own report.
Artifact & serving — one sealed bundle, verified before binding
The ~257 MB transformer bundle is deliberately untracked. make serve and make demo verify it by checksum before binding a port, then serve typed /health and /v1/predict. The demo proves one accept (activate_my_card at 0.5300) and one abstain (0.0413, intent=null) over a real socket. Interface contract
What these numbers do not support
- • A confidence score here is a ranking signal for abstention, not a probability of correctness.
- • The curated unsupported-request check is a behavioral check, not an out-of-distribution benchmark.
- • Descriptive for that machine, not a service level.
- • The curated unsupported-request fixture is a behavioral check, not an OOD benchmark.
- • Neither model's confidence is a probability of correctness.
- • The demo is local-only; the acceptance audit is evidenced by a fully evidenced local run and a degraded CI run — by declaration, not as a deficiency awaiting a fix.