All things data. One authority.
AI and training data · Brief

When three readers disagree, the label hides it

A new Arabic dataset keeps every reader's verdict, and shows what a single majority label throws away.

19 November 2026 · 1 min read

Ask three people whether a comment is sexist and they will not always agree. That is normal, because the judgement depends on who you are and where you grew up. Most datasets still squash the answers into one majority label and throw the disagreement away.

Researchers Alaa Bazaid and Arkaitz Zubiaga built MuDAS, a set of YouTube comments in Saudi, Egyptian and Moroccan Arabic. Each comment was labelled by three native speakers of its dialect, and every individual verdict was kept.

They found that models make more mistakes on comments where the annotators disagreed. Models with similar overall scores also failed in different ways: some flagged harmless religious advice, others missed sexism that was implied. A single majority-label score hid all of that.

For anyone who owns or buys labelled data, the lesson is plain. Keep the individual judgements, not just the final label. A dataset that records where people disagreed is worth more, because it shows a buyer where a model trained on it is likely to go wrong.

Looking for data the internet does not have?

Tell us what you are training and we will show you what owners are offering.

Start on abura.io
Sources

How these stories are written: each is drafted with AI from its primary source (the paper, filing, ruling or announcement itself), which the story links, and every figure links to where it came from. Each is held before publishing so our editor can stop it, and a story that names a person, accuses anyone, or covers a court case or a regulator's action goes live only when the editor says so. Other outlets' reporting is linked, never rewritten.

Get the weekly issue

One email a week once the issue starts, only the desks you pick. Leave whenever you like.

Your desks
Or follow topics
Your topics

We use your email only to send the issue. Your rights.

More from AI and training data

The data desks

Value
Protect
Share
Grow

Desks marked ↗ lead to related partner sites.