Three statistical mistakes account for most of the results that fall apart under review, and all three are easy to avoid once named. None of them are specific to Arabic, but Arabic NLP work hits them disproportionately because class imbalance and dialect slices multiply the number of comparisons being made.
1. Macro-F1 has no closed-form confidence interval
People report macro-F1 with a ± next to it, computed as though it were a proportion. It is not. Macro-F1 is an average of per-class F1 scores, each of which is itself a harmonic mean of two ratios. There is no standard error formula that applies.
Bootstrap it. Resample your test set with replacement, recompute macro-F1 on each resample, and take the 2.5th and 97.5th percentiles.
One detail that matters more than it looks: resample conversations, not turns. Turns within a conversation are correlated — same speaker, same topic, same dialect, same generator draw. Treating them as independent units shrinks your interval by a factor that has nothing to do with your evidence.
2. Two-arm comparisons are paired
The most common setup in this area is a comparison: a single-turn baseline against a context-aware arm, both evaluated on the same test set.
The wrong move is to compute a confidence interval for each arm, notice they overlap, and conclude there is no difference. Overlapping intervals do not mean no difference when the two arms are evaluated on the same items.
Both models saw the same turns. That pairing is information, and throwing it away costs you real statistical power. Use McNemar's test on the disagreement counts: the cases where arm A was right and arm B wrong, against the reverse. That is what the comparison actually rests on.
| Question | Wrong tool | Right tool |
|---|---|---|
| Confidence in one model's macro-F1 | Normal approximation | Bootstrap over conversations |
| Does arm B beat arm A on the same set? | Compare overlapping CIs | McNemar on paired disagreements |
| Is my per-class recall estimate stable? | Point estimate | Wilson interval on class count |
3. Pass rates measured at n=20 are noise
This one comes up constantly when calibrating a generation pipeline. You run a model against your quality gate on twenty rows, get 18 passes, and record a 90% pass rate.
Here is real measured variation on identical configuration, same model, same prompt, same gate:
| Sample | Result |
|---|---|
| Run 1 | 12 / 12 (100%) |
| Run 2 | 24 / 24 (100%) |
| Run 3 | 21 / 24 (88%) |
And a second model across three samples: 33%, 67%, 92%.
The spread is not the model changing. It is what small-sample binomial variance looks like. A 90% pass rate at n=20 has a 95% interval of roughly 68–99% — wide enough that it cannot distinguish a good configuration from a mediocre one.
Budget from a pooled figure across runs, not a single sample, and quote the denominator every time you quote a rate. "95% (59/62)" is a claim someone can evaluate. "95%" is not.
The reporting checklist
- Macro-F1 with a bootstrap interval, resampled over conversations
- Per-class recall with Wilson intervals and the class count next to it
- Two-arm claims backed by McNemar, not overlapping intervals
- Every rate quoted as numerator/denominator
- The smallest class size in the test split, stated plainly
None of this makes a weak result strong. It makes a real result defensible, and it stops you announcing one that was never there.