A row that fails validation after its repair attempts is deleted, not hand-corrected. This is the rule that surprises people most, because deleting data you paid to generate feels wasteful. It is the cheaper option by a wide margin.
What hand-editing actually introduces
Suppose a conversation fails a check — the agent never named the brand, say — and someone opens the file and adds the brand name to one turn.
That edit is not neutral. It has a signature: a human, applying a rule, in a consistent place, in a consistent register. Do it across a few thousand rows and the corpus now contains a learnable pattern that has nothing to do with the phenomenon you were modelling.
Models are extremely good at finding exactly this kind of regularity. The edit is systematic, so it is easy to learn, and it is correlated with the very cases that were hardest to generate — which is to say, the interesting ones.
You have not fixed those rows. You have marked them.
The arithmetic
The case for editing is that generation costs money. Run the numbers and it collapses:
| Cost | |
|---|---|
| One generated row (measured, GLM-5.3-flash) | ~$0.0008 |
| Three repair attempts on a hard row | ~$0.002 |
| One minute of a person's attention | Several hundred rows' worth |
A row is worth less than a tenth of a cent. A human deciding whether to keep it costs more than regenerating it several times over. There is no budget in which hand-editing is the economical choice.
Repair is fine; editing is not
This is not an argument against the repair loop. There is an important difference between the two:
Repair sends the row back to the model with the specific complaint and lets it re-emit. The output is still model-generated, from the same distribution as everything else. Whatever bias it has, the rest of the corpus has too.
Editing puts a human in the output distribution for a non-random subset of rows. That is the problem — not that a human touched it, but that a human touched these particular rows and not the others.
So: three model attempts, then drop.
What to do with the dropped rows
Do not just discard them silently. A dropped row is a measurement.
- Tag it with which check it failed. A rejection-code histogram tells you where your recipe is weak. If 40% of failures are one validator, that validator is either miscalibrated or your prompt does not set the model up to satisfy it.
- Publish the drop rate. It belongs in the data card next to the row count. A corpus with a stated 5% drop rate is more trustworthy than one with no number, because the second one either did not measure or did not like the answer.
- Watch for a zero. If nothing is ever dropped, your gate is not gating. That is a bug report, not a success.
The version of this that is genuinely hard
The rule gets uncomfortable when a whole class fails disproportionately — say your rarest intent drops at three times the corpus rate. Dropping honestly will skew your label distribution exactly where you can least afford it.
The right response is still not to hand-edit. It is to generate more of that class and let the gate do its work, or to conclude that the class as defined is not expressible and fix the codebook. Both are real fixes. Editing is a way of not finding out.