A price tag for every page an AI crawler reads
Cloudflare lets website owners charge AI crawlers by the request, reviving an old web code for payment.
Picture made for ABURA Bureau.
On 1 July 2025 Cloudflare, which sits in front of a large share of the world's websites, introduced pay per crawl, a private beta that lets a site owner set a price for AI crawlers. Crawler by crawler, the owner chooses to allow free access, to charge, or to block.
A crawler that has not offered to pay gets back HTTP 402 Payment Required, with the price, a status code written into the web's rules decades ago and almost never used. Cloudflare acts as merchant of record: it records each paid request, charges the crawler and passes the earnings to the publisher. For now the price is one flat rate per request across a whole site, and the company calls it an early experiment, with different prices for different content, and for training versus other uses, still to come.
The significance is the default. Until now a website's choice was mostly to leave the door open or shut it. A third option, open at a price, turns a site's pages into something sold by the unit. Owners of private data have always had that choice in principle. The public web is now being given the tools to act on it.
