If your accuracy jumped three points for no clear reason, the first suspect is that your model already saw the test set. Not literally the same rows — near duplicates. One in train, one in validation, differing by a word.
This is the most common defect in synthetic corpora and the hardest to notice, because it has exactly one symptom and that symptom is good news.
Exact duplicates are not the problem
Every pipeline drops exact duplicates; it is a hash lookup. Synthetic data rarely produces them anyway, because the generator varies surface form freely.
What it produces instead are rows like this:
train[8821] "طلبي تأخر ثلاث أيام وما وصل"
val[112] "طلبي تأخر ثلاثة أيام وما وصلني"
Different strings. Same conversation. A model that memorised the first will get the second right for the wrong reason, and your validation number is now measuring memorisation.
Why synthetic data is especially exposed
A recipe-driven generator draws from a finite grid of dimensions. Two rows that land on the same combination — same dialect, same sector, same topic, same sentiment — are supposed to be similar. That is the taxonomy working.
It also means near-duplicates are not a rare accident. They are a structural consequence of sampling a finite space many times, and their rate climbs as the corpus grows.
Deduplicate before you split, never after
This is the part people get backwards.
The instinct is to split first — it is the first thing in every tutorial — and then deduplicate within each split. That removes redundancy inside train and inside validation, and leaves every cross-split pair exactly where it was. Which is the only kind that affects your metric.
The correct order:
- Compute a similarity signature for every row in the whole corpus
- Cluster rows above your similarity threshold
- Assign whole clusters to a split, never individual rows
- Then split
Rows that are near-duplicates of each other end up on the same side of the wall by construction. There is no cross-split pair left to find.
Choosing a threshold, and what it costs
MinHash over word-level shingles at around 0.85 similarity is a reasonable default for Arabic. Two things to know:
Use word shingles, not character shingles. Character n-grams over text generated from a shared prompt template match heavily on boilerplate — the politeness formulas, the brand name, the structure the template imposed. You get a high similarity score driven by the parts every row shares, which is precisely not what you want to measure.
A threshold that removes nothing is misconfigured. If your dedup pass reports zero removals on a synthetic corpus, the threshold is too high or the signature is wrong. Zero is a bug report.
Publish the number
Put removed-row counts in your data card, per split. Two reasons:
- A reader can judge whether your evaluation is trustworthy without rerunning it.
- A zero in that field is a visible claim rather than a silent omission, which means someone will check it.
A corpus that states "we removed 1,240 rows for cross-split similarity above 0.85" is more credible than a larger corpus that says nothing — and the difference costs you one line.
What this does not fix
Cluster-aware splitting stops memorisation across the wall. It does not make your test set representative. If your taxonomy over-samples one dialect, both splits inherit the skew and your metric is still optimistic about the real world.
Leakage and representativeness are separate problems. Fixing the first is mechanical. Fixing the second means changing what you generate.