Some turns cannot be classified in isolation, and they are the only turns that can prove a context-aware model is worth building. If your dataset does not mark them, your context arm and your baseline will score almost identically and you will conclude, wrongly, that context does not help.
The phenomenon
In real support conversations, people drop what has already been established:
Customer: How much is the black abaya? Agent: 240 riyals, and it's in stock in all sizes. Customer: And the embroidered one?
That third turn is a price inquiry. Nothing in it says so. There is no price word, no currency, no product attribute — just a referring expression and an ellipsis.
Arabic does this heavily. Pro-drop, clitic pronouns and topic continuity mean the dropped material is often the entire content-bearing part of the sentence.
Why it breaks the obvious approach
A single-turn classifier sees والمطرزة؟ and has nothing to work with. But here
is the trap: so does your keyword-based validator.
If you are generating labelled data and you check that each labelled turn contains vocabulary matching its label, elliptical turns fail that check every time. The natural fix — make the model write a turn that does contain price vocabulary — destroys the exact phenomenon you wanted to capture. You end up with a corpus where every turn is self-contained, and a context-aware model has nothing to learn from.
Mark them explicitly
The fix is to carry a flag alongside the label:
| Field | Meaning |
|---|---|
label |
The intent, e.g. price_inquiry |
context_dependent |
True when the turn's meaning requires prior turns |
context_type |
What kind of dependency — ellipsis, coreference, continuation |
Then make your lexicon validator skip turns flagged context-dependent. An elliptical turn cannot contain its own label's vocabulary; that absence is the phenomenon, not a defect.
This one exemption is the difference between a dataset that can answer "does context help?" and one that cannot.
What it buys you at evaluation time
With the flag present, you can report something far more interesting than an overall score:
- Accuracy on self-contained turns — where a single-turn baseline should be competitive
- Accuracy on context-dependent turns — where it should collapse
- The gap between arms on each subset
If your context arm beats the baseline overall by two points but by fifteen points on context-dependent turns, you have a finding. Without the flag you have the two points, and no way to argue they are not noise.
How much of the corpus should be context-dependent?
Enough that the subset is measurable, which means enough instances in the test split to compute a stable per-class number — a few hundred at minimum.
Resist the temptation to make it the majority. Real conversations are mostly self-contained with elliptical turns clustered after an initial establishing exchange. A corpus that is 60% elliptical is not a harder benchmark, it is an unrealistic one, and a model trained on it will over-predict context dependence in production.
The general lesson
The turns that are hardest for your validator are frequently the ones carrying the signal you are selling. When a check keeps failing on a particular kind of row, the question to ask before tightening the generator is whether the check is rejecting a phenomenon you actually want.