Default to JSONL. Offer Parquet for scale. Treat CSV as a compatibility concession. The reasoning is short, but there is one failure mode in columnar export that is worth knowing before it bites.
The quick comparison
| JSONL | Parquet | CSV | |
|---|---|---|---|
| Nested conversation turns | Natural | Natural (list of structs) | Requires flattening |
| Optional / missing fields | Just omit the key | Needs a declared schema | Empty cell, ambiguous |
| Streaming row by row | Yes | Awkward | Yes |
| Column subset reads | No | Yes | No |
| Human-inspectable | Yes | No | Yes |
| Size on disk | Large | Small | Medium |
| Arabic text safety | Good with UTF-8 | Good | Fragile |
Why JSONL is the right default
Training data for conversational tasks is nested — a row contains a list of turns, each with a role and content, sometimes with per-turn annotations. JSONL represents that without ceremony, one object per line.
The property that matters most for evolving datasets: a JSON object may simply
omit a key. An agent turn that carries no intent label just has no label
field. Nothing needs to be declared, and adding a field later does not invalidate
existing rows.
It also streams. You can read a 50,000-row file line by line in a few lines of
code, and head -1 shows you the schema.
Why CSV causes trouble with Arabic specifically
CSV is fine for a flat label table and painful for anything else.
Flattening destroys the structure. A conversation becomes either one row per turn — losing the grouping unless you carry a conversation id and trust the order — or one row with numbered columns, which caps your turn count.
Quoting and Arabic punctuation interact badly. Arabic text contains commas
(،), and while that is not the ASCII comma, real text mixes both. Any row with
an embedded newline or quote depends entirely on the writer and reader agreeing on
escaping, and they frequently do not.
Excel will mangle it. If anyone opens your CSV in a spreadsheet and saves, encoding and RTL display marks can change under them. This happens more often than you would like.
Ship CSV when a consumer's tooling requires it, and only for flat data.
Parquet, and the trap
Parquet is the right answer when consumers want column subsets from a large
corpus, or when the file goes into a data warehouse. It keeps the conversation as
a nested list of structs, which is the shape Hugging Face datasets loads without
conversion.
The trap: Parquet requires a declared schema, and inferring it per shard breaks concatenation.
Consider a corpus where only customer turns carry a context_dependent boolean.
Export it in two shards:
- Shard A happens to contain no rows with that field → the struct is inferred without it, or with a null type
- Shard B contains them → the struct is inferred with a boolean field
The two shards now have different schemas and refuse to concatenate on the consumer's side. They did nothing wrong; your exporter inferred types from data instead of declaring them.
The fix is to declare the column types explicitly in your recipe or export config, per annotation key, and write that schema into every shard whether or not the shard has values for it. Types should be a property of the dataset definition, not of whichever rows landed in a file.
What to actually offer
- JSONL as the canonical download
- Parquet with an explicitly declared schema, for scale
- CSV on request, flat only
- Field selection at export, so a consumer is not stripping four columns by hand before training
That last point is small and disproportionately appreciated. Most people do not want your internal debug fields, and making them delete them is a bad first impression of the data.