ADR-0001: DistilBERT with Hugging Face Transformers and PyTorch
Accepted2026-07-31Context
The portfolio project must demonstrate modern NLP model work, PyTorch or TensorFlow, Hugging Face, reproducible evaluation, and local execution on an RTX 5060 Mobile with 24 GB RAM. It must also finish in one weekend. The improved method needs to be substantial enough to compare against a lexical baseline but small enough to fine-tune and serve locally.
Decision
Use `distilbert-base-uncased` through Hugging Face Transformers with PyTorch and a 77-class sequence-classification head. Use two epochs initially, sequence length 96, train batch size 16, validation macro-F1 for epoch selection, and one documented OOM fallback to batch size 8.
Alternatives considered · 5
- TF-IDF onlyFast and credible as a baseline, but insufficient to demonstrate Hugging Face/PyTorch fine-tuning.
- Frozen sentence embeddings plus logistic regressionLower implementation risk and valid as a fallback, but demonstrates less end-to-end transformer training.
- BERT-base, RoBERTa or DeBERTaPotentially stronger but larger/slower. Additional performance is not required to prove the engineering point.
- Small generative LLMWould make classification slower, harder to calibrate, and harder to evaluate without adding business value.
- TensorFlowTechnically valid, but the user’s local workflow and target portfolio benefit more from a straightforward PyTorch/Hugging Face path. Suppor…
Reconsideration trigger
Reconsider only if: - the checkpoint cannot load or make a forward pass on the target environment; - fine-tuning cannot produce a valid artifact by Hour 9 after the documented fallback; - measured latency or memory exceeds the hardware budget; - the target role explicitly requir…