Labels should be an input to generation, never an output of it. Draw the label first, ask the model to write text expressing it, then attach the label after parsing. The model should never be in a position to choose a label.
This sounds like a stylistic preference. It is not. Getting it backwards produces a corpus that is systematically mislabelled in a way no metric in your pipeline will report.
The failure, precisely
Synthetic generation pipelines almost always have a repair loop: generate a row, run validators, and if it fails, send the row back to the model along with the specific complaint and let it try again. This is the right design — it is how you get usable pass rates without hand-editing.
Now suppose the model emits labels. One of your validators checks that a turn
labelled price_inquiry actually contains price vocabulary. A row fails it, and
the complaint goes back:
Turn 5 is labelled
price_inquirybut contains no price vocabulary.
There are two ways for the model to satisfy that complaint. It can rewrite the turn to talk about price. Or it can change the label to something the turn already matches.
The second is cheaper, and the model will find it. Nothing in the loop prefers one repair over the other; both make the validator pass.
Why nothing catches it
Run that at scale and the corpus drifts into a specific shape: every turn is labelled with whatever your validator's regex would have guessed. Your label space has quietly been replaced by your keyword lists.
Meanwhile:
- Pass rates go up, because the model found the easy repair.
- Validator failures go down, for the same reason.
- Inter-annotator agreement on a sample may even look fine, because the labels are internally consistent with the text — they were fitted to it.
The only thing that has broken is the thing you were selling: the label carries no information the text did not already carry. A model trained on it learns your regexes.
The fix is structural, not a prompt instruction
Telling the model "do not change labels" does not work, because the repair loop's whole purpose is to let it rewrite its output. The fix is to make label emission impossible:
- Sample the label sequence before generation. An ordered arc — one label per turn you intend to annotate — is drawn from the taxonomy at planning time.
- Prompt for expression, not classification. The instruction is "write a customer turn that expresses asking about price", not "write a turn and label it".
- Whitelist the parser. When you parse the model's reply into turns, accept
only
roleandcontent. Anything else the model emits is discarded. - Zip the labels on afterwards, positionally, from the arc you drew.
Step 3 is load-bearing and looks like dead code to anyone reading it later. A two-key whitelist in a parser invites a well-meaning "fix" that turns it into a passthrough. Comment it accordingly.
The consequence you have to accept
With the label fixed, repair pressure can only flow into the text. That means a higher failure rate — the model genuinely has to write a turn that expresses the commanded label, and sometimes it cannot. Rows get dropped.
That is the correct trade. A dropped row costs a fraction of a cent. A systematically mislabelled corpus costs you the project, and you find out at evaluation time.
How to verify it empirically
Do not take the design on trust. Before scaling, confirm two things on a pilot:
- Label distribution matches the commanded arc, not the distribution a keyword classifier would produce over the same text.
- A native reader agrees that the turn expresses its commanded label, on a sample, blind to the label. Target something like 80% or better before you scale.
If the second number is low, your codebook is asking for distinctions the model cannot express — which is a finding worth having before you spend anything.