Europe's privacy regulators tell AI scrapers to stay away from sites that say no
As consultation on their web-scraping guidelines closes, Europe's data protection authorities set out what AI developers must do before they collect from the open web.
Picture made for ABUREAU.
Public comment on the European Data Protection Board's guidelines on web scraping for generative AI closed on 30 October 2026. The guidelines, adopted in July, set out how the GDPR applies when AI developers collect personal data from the web, whether they scrape it themselves, hire someone to, or reuse a data set someone else scraped.
The Board's summary for developers is direct. Avoid websites that block scraping with technical measures, such as robots.txt or ai.txt files, logins or CAPTCHA. Leave out sites used mainly by children and sites holding sensitive data. Scrape only from reliable sources, record when, and check the data before training. Before starting, consider synthetic data, set precise collection criteria and keep an inventory of what was taken.
The Board also reminds developers that sensitive data stays prohibited in principle, with no general exemption for training.
For owners, the guidelines put weight on a signal they already control. A site that says no to crawlers is now one a careful developer should skip. That makes data offered openly, under a licence and with personal data taken out, the easier route for anyone who needs it.
