Cloudflare announced in July 2026 that AI companies have until September 15 to separate web crawlers used for traditional search from those used for AI training and agents — or face automatic blocks on a wide range of publisher sites.
Starting September 15, 2026, Cloudflare’s default settings will block “mixed-use” crawlers from any pages that host ads. The change applies to new Cloudflare customers, new sites set up by existing customers, and all existing free-tier customers. Site owners can adjust their settings to override the default.
Cloudflare co-founder and CEO Matthew Prince tied the move to a recent milestone: bots have surpassed human traffic online for the first time, ahead of when that shift was expected to occur. “Now that the majority of traffic on the Internet is non-human, we must go further and act faster so that a sustainable ecosystem can emerge,” Prince said in his announcement.
The policy targets crawlers that blend search, agent use, and AI training into a single tool. Cloudflare specifically referenced the “world’s largest search engine” — an apparent reference to Google — saying it has access to roughly twice as much information as other AI companies because it makes it difficult for site owners to remain discoverable without also having their content used for AI purposes. Google has previously noted that it offers a separate opt-out tool called Google Extended, though its flagship Googlebot still crawls for Search, including AI features.
Alongside the crawler policy, Cloudflare is expanding its existing Pay Per Crawl marketplace into a “Pay Per Use” model, which would allow publishers to charge AI companies when their content generates value, not just when it is fetched. Cloudflare data indicates that over 50% of crawl traffic from AI crawlers is spent re-fetching unchanged pages, suggesting the change could also reduce unnecessary bandwidth use.
Cloudflare is initially partnering with Ceramic.ai and You.com to implement the Pay Per Use model. Publishers who opt in are paid when their content appears in Ceramic’s AI search results or when You.com accesses their premium content. Other AI companies can adapt the model to their own workflows, Cloudflare says.
Source: TechCrunch