S05.3 — Evaluate the unsupported-request fixture separately

Planning / historicaltaskdelivery

Historical

Planning material. It summarizes and links to the authoritative specification and the current measured state — it does not replace either, and it is not current engineering truth.

Migration identity#

Canonical identifier: S05.3 (GitHub issue type: subtask).
Legacy identifier: C05.3 (retained as old_identifier).
Parent epic: E05.
Milestone: M1 — IntentGuard Weekend MVP.

Objective#

Create a small declared unsupported-request fixture and report abstention behavior separately from BANKING77 metrics.

Rationale#

The abstention interface needs focused evidence without pretending a tiny fixture is a broad OOD benchmark.

Parent identifier#

E05.

Source task#

T-005.

Traceability#

Primary: FR-009, AC-012.

Prerequisites#

Persisted threshold and evaluation/report schemas complete; fixture content reviewed for privacy and representativeness limits.

Likely files#

tests/fixtures/unsupported_requests.jsonl, src/intentguard/evaluation.py, scripts/evaluate.py, tests/unit/test_eval_regression.py, reports/.

Implementation boundary#

Fixed local fixture used only for a separately labelled abstention summary; never merged into test accuracy/macro-F1.

MUST scope#

Fixture schema, provenance, expected category rationale, abstention counts/rate, sample-level output, and limitations.

Explicit non-goals#

OOD benchmark claim, threshold tuning, multilingual coverage, adversarial study, or production traffic simulation.

Acceptance criteria#

Every fixture row is evaluated once, results are separate from BANKING77, report language is bounded, and no fixture label influences threshold selection.

Validation commands#

make evaluate && uv run pytest tests/unit/test_eval_regression.py -q.

Expected evidence#

Versioned fixture, separate report section/file, counts/rate, model/artifact ID, and limitation statement.

Fallback and status consequence#

Absent or synthetic-only API behavior leaves FR-009 Planned/Partial and fails strict MVP.

Stop condition#

Stop for privacy-sensitive text, ambiguous fixture provenance, mixed benchmark metrics, or selection leakage.

Definition of ready#

Fixture purpose, content boundary, and report wording are reviewed.

Definition of done#

FR-009 and AC-012 have measured, separately labelled evidence without overclaiming.

Labels#

type:subtask, priority:MUST, area:evaluation

Estimate#

0.5 hour.