A tool that cleans health records for AI is put to the test
A new benchmark checks a chat agent that turns raw patient records into datasets ready for AI, and reports where it falls short.
Picture made for ABUREAU.
Raw health records are messy, and cleaning them takes much of the work before any AI can use them. A new paper, Benchmark of THReD, tests an agent that does the cleaning when asked in plain language, and only after the user confirms each step.
On the authors' own benchmark of 141 tasks the agent completed 92.7% on the first answer. When the data profile was taken away, completion fell to 58.0%. On real records from two chronic-disease groups, a separate check found 99.7 to 99.8% of vital-sign labels were inferred correctly.
The authors are open about limits. The agent test used synthetic tables, and the full paper says only one model was evaluated and that its checks were written by the authors. It also says protected columns such as identifiers and dates are not sent to the model.
For people who own or use data, preparation is where value is made or lost. A dataset is only as useful as the proof that it was cleaned the same way every time. The code and results are on Zenodo, so others can repeat the test before relying on it.
