Normalize for matching, ship the original. That one rule resolves most Arabic preprocessing arguments, and it is the opposite of what most pipelines do by default.
Arabic normalization is not a single step. It is a chain of independent decisions, each of which trades recall for fidelity, and each of which should be made separately.
The specific decisions
| Decision | What folding gains | What folding costs |
|---|---|---|
| Hamza forms (أ إ آ → ا) | Matches user input that omits hamza, which is most of it | Loses a real distinction; some minimal pairs collapse |
| Taa marbuta (ة → ه) | Matches informal typing | Erases a feminine marker that carries grammatical information |
| Alef maqsura (ى → ي) | Matches common substitution | Collapses a genuine final-form distinction |
| Diacritics (تشكيل) | Removes inconsistent decoration | Destroys disambiguation where it was deliberate |
| Tatweel (ـــ) | Removes pure decoration | Nothing. Always strip it. |
Only the last one is unambiguous. Tatweel is an elongation character with no semantic content; nobody has ever needed it preserved.
Why the default is wrong
The default in most pipelines is to normalize aggressively and store the result. It is attractive because it makes everything downstream simpler: fewer distinct tokens, better vocabulary coverage, higher match rates.
The problem is that it is lossy and one-way. Once you have folded أ into ا in the stored text, nobody can recover which it was. Every consumer of your dataset inherits your decision, including consumers whose task depends on the distinction you removed.
Meanwhile the reverse is trivial. Anyone who wants harsher normalization can apply it in one line. Nobody can un-apply yours.
The asymmetry is the whole argument
Normalizing is one line for a consumer. Un-normalizing is impossible.
So the burden of proof sits with folding, not with preserving. A dataset should ship text as written, with normalization available as a derived field.
In practice that means:
- Store the original text as the canonical
contentfield - Normalize into a parallel field used for search, deduplication and matching
- Document exactly which transformations the parallel field applies
Your files are slightly larger. That is the entire cost.
Where normalization genuinely belongs
Some places the folded form is the right one to use, and none of them require destroying the original:
Deduplication. Two rows differing only in hamza spelling are duplicates for any purpose you care about. Compute similarity signatures over normalized text.
Lexicon checks. If you are validating that a turn contains price vocabulary, match against normalized forms or you will miss the hamza-less spellings that dominate real typing.
Retrieval. Search should be forgiving. Index normalized, display original.
Never for training text. The model should see what people actually write, including the inconsistency. That inconsistency is a property of the language as used, and a model that has never seen it will be brittle when it arrives.
Diacritics are a separate question
Treat تشكيل differently from the letter-form decisions, because its presence is not random. If diacritics appear on a word in your source, they are usually there to disambiguate — someone chose to mark them.
The rule that works: keep diacritics where they are meaningful, strip them where they are arbitrary. In practice, for generated data, you decide this at generation time by instructing the model rather than post-processing, since only the generator knows whether the marking was deliberate.
A short checklist
- Strip tatweel unconditionally
- Keep the original text as canonical
- Normalize into a separate field, and say which rules you applied
- Deduplicate and lexicon-match on the normalized field
- Train on the original