When three readers disagree, the label hides it
A new Arabic dataset keeps every reader's verdict, and shows what a single majority label throws away.
Ask three people whether a comment is sexist and they will not always agree. That is normal, because the judgement depends on who you are and where you grew up. Most datasets still squash the answers into one majority label and throw the disagreement away.
Researchers Alaa Bazaid and Arkaitz Zubiaga built MuDAS, a set of YouTube comments in Saudi, Egyptian and Moroccan Arabic. Each comment was labelled by three native speakers of its dialect, and every individual verdict was kept.
They found that models make more mistakes on comments where the annotators disagreed. Models with similar overall scores also failed in different ways: some flagged harmless religious advice, others missed sexism that was implied. A single majority-label score hid all of that.
For anyone who owns or buys labelled data, the lesson is plain. Keep the individual judgements, not just the final label. A dataset that records where people disagreed is worth more, because it shows a buyer where a model trained on it is likely to go wrong.
