One call per turn is the single most expensive mistake in an annotation pipeline, and it is the default shape everyone writes first. On a 44,000-turn corpus it costs roughly $360. Batched properly, the identical work costs around $11.
The difference is not the model. It is what you resend.
Where the money actually goes
A blind relabelling pass — asking a model to independently assign a label to each turn so you can measure agreement against your own labels — has a natural naive shape:
for turn in corpus:
call(model, system=codebook_15_labels, user=turn.text)
Every one of those calls resends the entire codebook. For a 15-label scheme with definitions and examples, that system block is easily 800–1,500 tokens. Multiply by 44,000 and you have sent the same menu forty-four thousand times, at frontier model prices, to classify text that is often under twenty words.
| Approach | Calls | What gets resent | Est. cost (44k turns) |
|---|---|---|---|
| One call per turn, frontier model | 44,000 | Full codebook, every call | ~$360 |
| One call per turn, cached system block | 44,000 | Codebook cached | ~$90 |
| One call per conversation, cheap model, cached menu | ~11,000 | Codebook cached, 4 turns per call | ~$11 |
For context, generating that entire corpus costs about $17. A naively written labelling pass costs twenty times the thing it is checking.
Three changes, in order of impact
Batch by conversation, not by turn. Ask for a label per customer turn in one call and return an array. A four-turn conversation becomes one request instead of four. This alone removes three quarters of your call count, and the model sees the turns in context — which is usually what you wanted anyway.
Cache the codebook. Every major provider now supports caching a static system prefix. Your label definitions do not change between calls; they are the textbook case. Put the menu in a cached block and you stop paying full input price for it.
Use a different family, not a better one. A blind relabelling pass exists to produce an independent opinion. If you label with the same model family that generated the text, agreement is inflated by shared priors — you are measuring self-consistency, not label quality. A cheap model from a different family is both more correct methodologically and an order of magnitude cheaper.
The trap in "use a better model for labelling"
It sounds like rigour and is usually the opposite. Frontier models are expensive precisely because they reason, and a closed-set classification over short text does not need reasoning — it needs consistency. What you want from a relabeller is a stable, cheap second opinion you can compute agreement against.
If agreement between your labels and the relabeller is high, that is evidence. If it is low, the relabeller has told you which classes your codebook fails to separate, which is worth far more than a slightly more accurate label.
Budget it against the right baseline
The number to compare a labelling pass against is not "is $360 affordable". It is "what did the data cost, and does the check cost more than the thing it checks". A verification step that dominates your budget is a design error, not a cost of doing business.