S05.2 — Measure calibration, selective prediction, and latency

Planning / historicaltaskdelivery

Historical

Planning material. It summarizes and links to the authoritative specification and the current measured state — it does not replace either, and it is not current engineering truth.

Migration identity#

Canonical identifier: S05.2 (GitHub issue type: subtask).
Legacy identifier: C05.2 (retained as old_identifier).
Parent epic: E05.
Milestone: M1 — IntentGuard Weekend MVP.

Objective#

Apply the persisted threshold and report ECE, coverage, selective risk, abstention, and latency with explicit conventions.

Rationale#

Confidence-aware classification requires evidence beyond top-line accuracy.

Parent identifier#

E05.

Source task#

T-005.

Traceability#

Primary: NFR-006. Secondary: FR-004, FR-005, AC-011.

Prerequisites#

C05.1 report identity fixed; ECE bins, numerical tolerances, warm-up/sample protocol, and latency environment resolved.

Likely files#

src/intentguard/evaluation.py, src/intentguard/metrics.py, tests/unit/test_metrics.py, tests/unit/test_threshold.py, reports/.

Implementation boundary#

Load and apply the artifact threshold exactly; no re-selection, temperature scaling, or test-label tuning.

MUST scope#

ECE convention, confidence bounds, accepted/abstained partition, coverage, selective risk, latency samples/statistics, and environment metadata.

Explicit non-goals#

Temperature scaling, per-class thresholds, SLO claims, load testing, or GPU latency extrapolation.

Acceptance criteria#

Metrics are deterministic within declared tolerance, rows partition completely, threshold equals artifact metadata, and latency protocol/environment are reported.

Validation commands#

make evaluate && uv run pytest tests/unit/test_metrics.py tests/unit/test_threshold.py -q.

Expected evidence#

ECE bins, coverage/risk values, counts, threshold identity, latency distribution, warm-up/sample settings, hardware/software metadata.

Fallback and status consequence#

CPU-only measurement is valid and labelled. Missing calibration/selective/latency evidence means not Measured and strict MVP fails.

Stop condition#

Stop for undefined binning/tolerance, threshold mismatch, row loss, test tuning, or misleading timing protocol.

Definition of ready#

Metric and timing conventions are decided before test results are inspected.

Definition of done#

NFR-006 has reproducible evidence tied to the immutable artifact; its secondary AC-011 relationship is recorded without changing AC-011 primary ownership.

Labels#

type:subtask, priority:MUST, area:evaluation

Estimate#

0.75 hour.