Synthetic data has a size past which each additional row makes the corpus narrower rather than broader. For a taxonomy-driven generator with a finite seed grid, that point arrives around 25,000 rows. It is worth understanding why, because the instinct to just generate more is strong and the failure is silent.
The taxonomy is the variation, not the model
A recipe-driven generator draws each row from a combination of dimensions — dialect, sector, sentiment, topic, brand, and so on. The number of genuinely distinct rows it can produce is bounded by that grid, not by how many times you call the model.
Once you have sampled the grid densely, further generation produces rows that differ from existing ones in wording but not in structure. A near-duplicate filter is very good at spotting exactly that.
What the filter does at scale
Run MinHash deduplication at a 0.85 similarity threshold over word-level shingles and something counterintuitive happens as the corpus grows:
| Corpus size | What the filter removes |
|---|---|
| Early rows | Genuine accidental repeats |
| Mid corpus | Rows that reuse a combination already covered |
| Past saturation | Rows that are varied in surface form but land near an existing cluster |
The rows most likely to be caught late are the ones that explore an unusual phrasing of a well-covered combination — which is to say, some of the most interesting rows you generated. What survives is the dense centre.
So the corpus keeps growing in row count while shrinking in coverage. Nothing in the pipeline reports this as an error. Your dedup rate creeps up, which looks like the filter working harder, and in a sense it is.
Four other ceilings arrive at the same time
Review coverage collapses. The number of rows a human will actually read is fixed. Every row you add divides that constant.
Corpus-level constraints get harder. If you enforce a target label distribution, or a cap on how often a phrase may appear across the corpus, those constraints tighten as the corpus grows. More rows means more reject-and-regenerate cycles for zero scientific gain.
Compute, not money, becomes the binding constraint. Generation is cheap. Fine-tuning twelve to twenty times over a corpus four times larger is not.
The label ceiling does not move. If your annotation scheme tops out at 0.90 agreement, a corpus twice the size still tops out at 0.90 agreement. You have doubled the rows under the same ceiling.
How to tell where your saturation point is
You do not need to guess. Track two numbers as you generate:
- Rejection rate from the near-duplicate filter, per thousand rows. When it starts climbing steadily rather than sitting flat, you are approaching saturation.
- Distinct taxonomy combinations covered, as a fraction of the grid. When this flattens while row count keeps rising, every new row is a re-draw.
When both curves bend, stop. If you need a larger corpus after that, the answer is to widen the taxonomy — add a dialect, add a sector, add topics — not to run the same grid for longer.
The honest version of "our dataset is 44,000 rows"
A defensible size claim is not the biggest number you can produce. It is a number with the coverage statistics next to it: how much of the grid you touched, what your duplicate rate was, and what a learning curve says about whether more would have helped. That is a far stronger position than a larger corpus with none of those numbers attached.