Ask a large language model for a customer-support conversation in Saudi Arabic and you will get something that reads fluently and is wrong in ways that only show up at scale. The words are Arabic. The dialect is a blend. The support agent talks like the customer. And every third conversation resolves with the same stock phrase.
None of that is visible in a sample of five. It is extremely visible in a dataset of fifty thousand, which is the point at which someone is training on it.
This post covers the checks we run on every generated conversation before it enters a dataset, and why each one exists.
Dialect contamination is the obvious failure
The first problem is the easy one to describe and the easy one to catch. Models trained predominantly on Levantine and Egyptian Arabic leak those forms into text that is supposed to be Gulf Arabic.
A handful of words are reliable tells:
| Leaked form | Region | Saudi equivalent |
|---|---|---|
| بدي | Levantine | أبغى / أبي |
| شو | Levantine | وش / إيش |
| عايز | Egyptian | أبغى |
| مشان | Levantine | عشان |
| هلق | Levantine | الحين |
Any appearance of these fails the conversation outright. This is a regex, it costs nothing to run, and it catches a surprising share of otherwise-plausible output.
The harder failure: register
Dialect is not one axis. A Najdi customer and a Najdi support agent do not speak the same way, and conflating them is the mistake that makes synthetic support data read as fake to a native speaker.
The customer says أبيك, وش ذا, يا رجال. The agent never does — those are
too informal for someone representing a company. The agent has their own
register: طال عمرك, سم, أبشر, حقك علينا.
So the check is directional. For each dialect we hold two lists: markers the agent must draw from, and markers the agent must never use. A conversation where the agent mirrors the customer's informal speech fails, even though every individual word is correctly Najdi.
That directionality is the part a general-purpose model does not get right on its own, and it is not something you can prompt your way out of reliably — you have to check the output.
Caricature
The opposite failure is a model that has been told "use Najdi markers" and proceeds to use all of them, in every turn. Real speech is not evenly seasoned.
We cap marker density two ways: no single customer turn may carry three or more heavy dialect markers, and no conversation may exceed six across all customer turns. Both numbers came from reading output that felt wrong and working out why.
The same principle applies to hospitality phrases on the agent side. Saudi
service language is genuinely warm — أبشر, من عيوني, على راسي — but an
agent who says أبشر in all eight turns is a template, not a person. No tracked
phrase may appear more than twice in one conversation.
Domain coherence
Dialect can be perfect while the conversation is still unusable.
A digital wallet cannot open a bank account. A telecom operator cannot refund a food order. A government services platform cannot issue cashback. Models invent these capabilities constantly, because "offer the customer something" is a strong prior and the specific constraint is not.
We hold an explicit capability matrix per brand — what it is allowed to do and what it must never offer — and check the agent's turns against the forbidden list. We also require sector-appropriate vocabulary to actually appear: a telecom conversation with no mention of شبكة, باقة, شريحة or فاتورة is not really a telecom conversation.
Sequencing
The check that catches the most subtle failure: a support agent must verify the customer's identity before acting on their account.
Models routinely resolve the case and then ask for ID, because both events are
present in their training data and the ordering constraint is weak. So we find
the first agent turn containing a verification pattern (last four digits of the
ID, an OTP, an order number) and the first turn containing an account-access
pattern (شايف, حررت لك, رجعت المبلغ), and fail the conversation if the
second comes before the first.
This one matters beyond realism. If you are training a support model on this data, ordering is exactly the behaviour you are teaching.
What happens when a check fails
Each failure produces a human-readable message — Agent must ask for verification before resolving the case, not ERR_4012. Those messages are fed
back to the model along with its own failed output, and it regenerates. Up to
three attempts, then the row is marked failed with its rejection codes recorded.
Writing the messages as instructions rather than error codes is deliberate: they are read by the model, not only by us.
In practice this costs about 2.25 attempts per delivered row. That is the real price of the quality gate, and it is most of the generation cost.
Duplication
Passing every check still leaves one problem. Within a fixed taxonomy — four dialects, four sectors, four sentiments, seventeen topics — rows in the same cell can come out very similar. We inject per-row variation (names, amounts, reference numbers, opening style, outcome, conversation length) and then measure whether it worked, with MinHash over word shingles across the whole corpus.
Measuring matters more than the injection. Anyone can add random names; the question is whether the resulting rows are actually distinct, and that is an empirical question about the corpus you shipped.
Why this is worth writing down
A synthetic dataset is only as good as what you refused to ship. The generation step is one API call that anyone can make. The checks are the work — and they are the difference between a file of plausible Arabic text and a dataset someone can train on.