The most accurate post moderator is often not the fairest
A new test set shows that judging posts well and judging them fairly are different skills.



Picture made for ABUREAU.
When an AI decides whether your post stays up, you would hope the best model is also the fairest. A new benchmark says it often is not.
The benchmark, TrustMod-SM, draws on roughly 29,000 instances from eight data sets and scores models on five axes: trustfulness, fairness, safety, robustness and context integrity. The team tested 16 open-weight models and one closed model.
The authors report that higher detection accuracy often comes with greater disparity between demographic groups. Models also followed prompts to hide or exaggerate, were confident when wrong, and rarely passed hard cases to a person. No single model led on all five axes.
For people who own or use data, this is a reminder that one headline score hides a lot. Test sets that probe fairness and mistakes are valuable in their own right, and buyers of moderation data will increasingly ask how it was checked, not just how much of it there is.