A legal AI maker tests its agent without seeing client data
Legora's CTO explains how to improve an AI product when the data belongs to the customers' clients.
In a Latent Space interview, Legora's CTO describes a plain limit: the company cannot see the traces of what its agent does for a lawyer, because that material belongs to the law firms' clients.
The team works around it in three ways, as the show notes summarise. Engineers and lawyers build simulated test worlds modelled on customer work. Failures are paraphrased and grouped into roughly 10 to 20 types, and product metrics stand in for reading traces. The notes also mention anonymised examples as another route.
One example given is a citation problem that went from about 70% to 99% accuracy after a single release. Legora also runs on one cloud, largely for privacy reasons.
For people who own or use data, the point is that holding data and building on it are different things. An owner can set terms that keep raw material out of a builder's sight, and builders that cope with that limit may be the ones trusted with the most sensitive data.
