Igala gets open translation data that records dialect
A small sentence set for a Nigerian language shows why the mix of dialects in data changes results.
Picture made for ABUREAU.
Igala, a language of Nigeria, had no parallel text for machine translation. A new preprint by Etuh E. and Eli. A. Jiya builds a first open set and tags each sentence by dialect.
The set holds 1,137 English and Igala sentences, 899 of them used for training. Idah makes up 40.4 percent, while Ogugu is only 2.8 percent. A model trained on mixed dialects scored better overall than one trained only on Idah, which did notably worse on Ibaji.
The authors report a BLEU score of 11.66 for the mixed model, a modest baseline, and release the data under a Creative Commons Attribution licence. A separate record is held on Zenodo.
For people who own or use data, a small set can still carry value when it covers something no one else has. Labels such as dialect make the set more useful and let a buyer see what it covers and what it misses. Owners of language data should record that make-up from the start.
