Registry text alone carries some signal on which trials succeed
Language models read ten public fields per trial and beat the published benchmark, but not a trivial guess on every measure.
Picture made for ABUREAU.
Failed trials raise the cost of new medicines. A preprint asks how much the free text in public trial records says about the result. It uses the TOP benchmark of 12,477 trials, each turned into text from ten ClinicalTrials.gov fields.
Two biomedical language models were tried, in four set-ups, against the earlier HINT method. All four beat the published HINT scores on the two ranking measures, with PR-AUC gains of 10 to 20%. The best, a frozen 7-billion-parameter model on phase 3 trials, reached a PR-AUC of 0.901.
The authors are careful about the limits. F1 results were small and mixed, and no model, HINT included, clearly beat a trivial classifier that predicts success for every trial. Most differences between set-ups were under 0.03 and were not tested for significance. Simply reading the data, without retraining the model, did about as well as retraining it.
For people who own or use data, the lesson is that old public records still hold signal, and a cheap method may be enough to find it. The same paper is a reminder to compare any model with the laziest possible answer before paying for it.
