Here is a failure mode that does not announce itself. You generate fifty thousand conversations from a taxonomy of a thousand or so combinations. Every row passes your quality checks. Every row reads well. And roughly fifty rows per combination are near-restatements of each other, because the only thing varying inside a cell is a name and an amount.
The buyer finds this in an afternoon. You should find it first.
Why the taxonomy is not the variation
It is tempting to reason: four dialects times four sectors times four sentiments times seventeen topics is 1,088 distinct scenarios, so a 50k dataset has plenty of variety.
It does not follow. Each combination repeats about forty-six times, and the prompt for all forty-six is identical. Whatever variation exists comes from whatever you injected on top — and from sampling temperature, which is a weak and unreliable source of genuine diversity.
So variation has to be deliberate: customer name and age, agent name, brand, monetary amount, reference number, opening style, outcome, and the number of exchanges. Then the actual question: did it work?
Word shingles, not character shingles
The standard approach is MinHash with locality-sensitive hashing. Represent each document as a set of shingles, hash to a fixed-length signature, use LSH to find candidate pairs without comparing everything to everything.
The choice that matters is what a shingle is.
For English prose, character n-grams are a common default. For this corpus they are actively misleading. Every conversation is generated from a shared prompt template and shares a great deal of boilerplate structure — greetings, verification phrasing, closings. Character shingles match heavily on that scaffolding and report similarity that reflects the template rather than the content.
Word-level shingles of four consecutive words are a much better fit. They are long enough that shared boilerplate does not dominate, and short enough to catch genuine restatement.
Two normalisations before shingling: collapse whitespace and case-fold. Both are cheap and both prevent trivially different strings from reading as distinct.
Two layers
Exact duplicates are worth catching separately with a plain content hash. It is far cheaper than MinHash, it is exact, and a corpus with exact duplicates has a different problem from one with paraphrases — usually a bug in how rows are reserved or written, not a generation-quality issue.
Near duplicates then go through MinHash/LSH with a Jaccard threshold. We keep the default at 0.85 for the in-run check and scan the corpus at lower thresholds before shipping, because a stricter threshold is the more honest test.
Reading the number
The output is a duplicate rate, and the useful thing about it is the trend as you tighten the threshold.
A corpus that shows nothing at 0.85 and nothing at 0.6 has genuine variation. A corpus that is clean at 0.85 and lights up at 0.7 has rows that a human would call "basically the same conversation" — which is what a buyer will conclude too.
Run it at more than one threshold. A single number at a threshold you chose to make the number look good is not evidence.
Where the check belongs
In two places, doing different jobs.
During generation, an in-memory index over the current batch catches a model that has fallen into a groove and is producing variations on one theme right now. This is bounded work and adds negligible cost.
Before shipping, a corpus-wide scan over everything completed. This is the one that matters, because it is the check a technical buyer runs on the delivered file. Finding a problem here is unpleasant. Finding it after the invoice is worse.
An honest caveat
MinHash measures lexical overlap. It will not tell you that four hundred conversations follow the same narrative arc in different words — customer complains, agent apologises, agent verifies, agent resolves, customer thanks. That is structural repetition, it is real, and Jaccard similarity does not see it.
Catching that needs a different instrument: embedding the conversations and looking at the density of the resulting space, or simply reading a random sample of thirty and noticing they feel the same. The second method is unfashionable and works well.
Measuring lexical duplication is necessary. It is not sufficient, and it is worth being clear about which question your metric actually answers.