Fine-Tuning
Our model layer supports supervised fine-tuning when a repeated workload benefits from behavior that cannot be made sufficiently reliable through prompting, retrieval, structured-output constraints, or other lighter-weight adaptation.
When we consider it
Fine-tuning changes model behavior by optimizing against task-specific examples. We consider it when the target behavior is stable, measurable, and valuable enough to justify the additional dataset, evaluation, versioning, and deployment burden.
Typical candidates include repeated classification, extraction, highly regular output generation, domain-specific interpretation, and workloads where a smaller specialized model can replace a larger general-purpose model without falling below the required quality threshold.
Training path
A production fine-tuning path can include:
- Workload definition
- Training dataset
- Held-out evaluation set
- Base-model selection
- Training configuration
- Candidate checkpoint
- Regression + workload evaluation
- Approved deployment
The training result is not accepted because loss decreased. It is accepted only if the new model clears workload-specific evaluation and does not introduce unacceptable regressions.
What we measure
Depending on the workload, evaluation can include:
- Task accuracy
- Output consistency
- Structured-output validity
- Hallucination or error rate
- Latency
- Inference cost
- Edge-case behavior
- Regression against previously working cases
Dataset quality is usually the limiting dependency. Noisy labels, leakage, class imbalance, stale examples, or weak representation of edge cases can produce a model that appears specialized while becoming less reliable in production.
When we do not use it
Fine-tuning is not a substitute for knowledge architecture.
Changing company facts belong in retrieval, temporal state, or authoritative data sources. Exact calculations belong in deterministic software. Poor entity resolution should be fixed upstream rather than encoded into model behavior.
Fine-tuning also creates lifecycle obligations: model-version coupling, drift, retraining, deployment complexity, and rollback. We use it when the model itself is the limiting component and the measured production improvement is large enough to justify owning that additional lifecycle.
Production versioning
A fine-tuned checkpoint is treated as a versioned production artifact, not as a one-time training output.
The deployed state needs to preserve which base model, dataset, training configuration, evaluation suite, and runtime configuration produced the approved behavior. When any of those inputs change, the resulting system can change even if the public workload definition appears identical. That traceability supports controlled rollback when a new checkpoint performs worse than the prior version.
Drift is handled the same way: if the company’s data distribution or operating process moves materially, the training distribution may no longer represent production. Retraining is therefore a measured response to demonstrated drift or performance loss, not a scheduled ritual.