Cloudflare has launched a new feature called Pay Per Crawl, currently in private beta, which allows website owners to charge AI bots a fee each time they scrape or “crawl” their site for data. This initiative fundamentally changes how AI companies access web content by requiring explicit permission and payment from AI crawlers before they can scrape websites protected by Cloudflare.
Key points about the Pay Per Crawl feature:
-
Control for Website Owners: Website owners can choose to block AI crawlers entirely, allow them for free, or set a specific micropayment rate per crawl request. They can also apply different rates or permissions to individual AI crawlers.
-
Verification and Security: Cloudflare uses public key cryptography (Ed25519 algorithm) to verify that AI crawlers are legitimate and have paid for access, preventing malicious actors from impersonating paying bots.
-
Impact on AI Training: Since Cloudflare protects about 20% of global web traffic, this feature could significantly affect how AI models gather training data, as many sites will now require payment or explicit permission for scraping.
-
Supporting Publishers: The move aims to protect original content creators from losing traffic and ad revenue due to AI bots scraping content without compensation. Major publishers and media companies like Condé Nast, TIME, The Associated Press, and others have already embraced this permission-based model.
-
Future Plans: Cloudflare envisions evolving the model to allow dynamic pricing and more granular control over content access, potentially motivating AI companies to negotiate content deals tailored to their specific needs.
Cloudflare’s CEO Matthew Prince emphasized that this initiative is about safeguarding the future of a free and vibrant Internet by giving power back to content creators while still enabling AI innovation.
Cloudflare’s Pay Per Crawl is a pioneering marketplace and technical framework that enables websites to monetize AI crawling, ensuring that AI companies pay for the data they extract, thereby supporting sustainable content creation and fair compensation for publishers.