Introducing pay per crawl: enabling content owners to charge AI crawlers for access
Cloudflare now looking to charge AI crawlers for content access. This is intriguing, and I hope it works -- AI crawlers have been extremely abusive in their crawling practices. Unfortunately I don't have high hopes, as the AI companies have already shown themselves to be happy to disguise their traffic as legit user accesses, with faked user-agent strings and use of proxies.... but let's see
Tags: cloudflare ai scraping llms http web pay-per-crawl