Fine-Tuned Agents vs. Off-the-Shelf LLMs: A Perimeter Cost Audit

The decision between an off-the-shelf model call and a fine-tuned, self-hosted agent isn’t primarily a capability question anymore — general models are good enough for most tasks out of the box. It’s a perimeter and cost question, and most teams are auditing the wrong side of the ledger.

What off-the-shelf actually costs

Per-token API pricing looks cheap in a demo and compounds quickly at production volume, especially for high-frequency internal tools running thousands of calls a day. The hidden cost isn’t the invoice — it’s that every call sends your data outside your perimeter to a third party, which is a real liability for anything touching proprietary or customer data.

What fine-tuning actually costs

Upfront training cost, ongoing hosting infrastructure, and the engineering overhead of keeping a fine-tuned model current as your underlying data changes. That’s real money — but it’s money spent once, inside your own environment, rather than a recurring per-call tax paid to an external vendor forever.

The audit framework we actually use

Three questions, in order: does this workload touch data that can’t leave your perimeter under any circumstance — if yes, fine-tune, full stop. If not, what’s the projected call volume at 12 months — below a threshold, off-the-shelf wins on total cost of ownership; above it, self-hosted starts paying for itself. Finally, does the task need domain-specific behavior a general model consistently gets wrong — if so, fine-tuning buys accuracy no prompt engineering will reliably deliver.

Most teams skip straight to “which model is smarter” and never run this audit at all. The smarter model is rarely the expensive mistake. The wrong deployment perimeter is.