All things data. One authority.
Newsletters
Latest
Life sciences A curated set and shared test make protein-model building comparableSovereign and infrastructure India proposes that AI models travel to the data, not the other way roundData value The data week so far: 1 to 3 November 2026AI and training data Big AI companies back one signal for a website to say no to trainingAI and training data Switching off redundant parts of a language model cuts how much text it repeatsTechnology One aviation safety dataset built from 128 sourcesAI and training data A legal AI maker tests its agent without seeing client dataLife sciences A health-records AI model holds up in hospitals it was never trained onLost and found When a startup closes, its code and conversations now have buyersPolicy and law EU governments agree their line on simpler data rules, with AI and research in view
AI and training data · Brief

Big AI companies back one signal for a website to say no to training

One robots.txt token could replace separate rules for each company's crawler, though start dates differ and it covers training only.

4 November 2026 · 1 min read

On 30 October 2026, Google and three other large AI companies committed to honour one robots.txt signal, as PPC Land reports. A website can use it to say its content should not be used to train generative AI. Until now a site needed separate rules for each company's crawler.

The signal is one token. It does not control crawling, and a blanket rule for all crawlers does not count as a preference. It comes from an individual draft announced through the IETF's AI Preferences group, which has no formal standing.

Start dates differ. PPC Land lists support in early 2027 for Google and from 1 February 2027 for another company. Two have named no date.

For owners, this is a cheaper way to state a preference, but it is a preference, not a lock. Our earlier report on Europe's regulators and AI scrapers shows signals of this kind already carry weight in Europe.

Looking for data the internet does not have?

Tell us what you are training and we will show you what owners are offering.

Start on abura.io
Sources

Get the weekly issue

One email a week once the issue starts, only the desks you pick. Leave whenever you like.

Your desks

We use your email only to send the issue. Your rights.

More from AI and training data

AI and training data · Brief

Switching off redundant parts of a language model cuts how much text it repeats

A preprint shows a way to reduce word-for-word recall of training text without retraining or changing the data.

4 November 2026 · 2 min readAI and training data · Brief

A legal AI maker tests its agent without seeing client data

Legora's CTO explains how to improve an AI product when the data belongs to the customers' clients.

3 November 2026 · 1 min read
A locked wrought-iron gate in an old stone wall around a sunlit hillside vineyard, a small spider-like robot crawler stopped at the gate and turning away down the dusty trackAI and training data
AI and training data · Brief

Europe's privacy regulators tell AI scrapers to stay away from sites that say no

As consultation on their web-scraping guidelines closes, Europe's data protection authorities set out what AI developers must do before they collect from the open web.

1 November 2026 · 2 min read

The data desks

Value
Protect
Share
Grow

Desks marked ↗ lead to related partner sites.