S05.1 — Produce comparable baseline and transformer classification metrics

Planning / historicaltaskdelivery

Historical

Planning material. It summarizes and links to the authoritative specification and the current measured state — it does not replace either, and it is not current engineering truth.

Migration identity#

Canonical identifier: S05.1 (GitHub issue type: subtask).
Legacy identifier: C05.1 (retained as old_identifier).
Parent epic: E05.
Milestone: M1 — IntentGuard Weekend MVP.

Objective#

Evaluate both immutable artifacts on identical canonical test rows with shared accuracy and macro-F1 implementations.

Rationale#

Transformer improvement is credible only when data, labels, preprocessing boundaries, and metric definitions are comparable.

Parent identifier#

E05.

Source task#

T-005.

Traceability#

Primary: T-005, FR-005, AC-004, AC-011.

Prerequisites#

U03 and U04 artifacts pass reload; U02 test fingerprint matches both.

Likely files#

src/intentguard/evaluation.py, src/intentguard/metrics.py, scripts/evaluate.py, tests/unit/test_metrics.py, tests/unit/test_eval_regression.py.

Implementation boundary#

Read-only artifact evaluation; no fitting, checkpoint choice, or threshold selection.

MUST scope#

Shared row/label order, accuracy, macro-F1, artifact/data IDs, improvement comparison, and numerical tolerances.

Explicit non-goals#

Retuning, significance claims not specified, charts, confusion commentary, or external benchmarks.

Acceptance criteria#

One report proves identical test membership, gives both required metrics for both artifacts, and evaluates AC-004 with declared tolerance. A tiny committed regression fixture produces the hand-checked metric and threshold results required by AC-011.

Validation commands#

make evaluate && uv run pytest tests/unit/test_metrics.py tests/unit/test_eval_regression.py -q.

Expected evidence#

Machine-readable comparison, test fingerprint, metric versions, values, deltas, and hand-checked metric and threshold regression output.

Fallback and status consequence#

Missing or incomparable real artifacts makes U05 Blocked/Partial; synthetic results cannot satisfy AC-004.

Stop condition#

Stop for artifact/data mismatch, evaluation-time fitting, unknown tolerance, test leakage, or inconsistent labels.

Definition of ready#

Both artifact manifests and evaluation metric definitions are frozen.

Definition of done#

FR-005, AC-004, and AC-011 have measured, reproducible, comparable evidence.

Labels#

type:subtask, priority:MUST, area:evaluation

Estimate#

0.75 hour.