Cloudflare is now experimenting with tools that will allow content creators to charge a fee to AI crawlers to scrape their websites.
In a blog Tuesday, Cloudflare explained that its "pay-per-crawl" feature is currently in a private beta. A small number of publishers—including AdWeek, The Associated Press, The Atlantic, BuzzFeed, Fortune, Gannett, and Ars Technica owner Condé Nast—will participate in the experiment. Each publisher will be able to set their own prices that bots must pay before scraping content, Cloudflare said.
Matthew Prince, CEO of Cloudflare, said the feature would ensure that the Internet as we know it will survive "the age of AI."
Cloudflare, a cloud infrastructure provider that serves 20% of the web, announced Tuesday the launch of a new marketplace that reimagines the relationship between website owners and AI companies — ideally giving publishers greater control over their content. For the last year, Cloudflare has launched tools for publishers to address the rampant rise of AI […]
The major internet architecture provider Cloudflare will now block known AI web crawlers by default to prevent them from “accessing content without permission or compensation,” according to an announcement on Tuesday. With the change, Cloudflare will start asking new domain owners whether they want to allow AI scrapers, and will even let some publishers implement a “Pay Per Crawl” fee.
The Pay Per Crawl program will let publishers set a price for AI scrapers to access their content. AI companies can then view pricing and choose whether to register for the “Pay Per Crawl” fee or turn away. This is only available for “a group of some of the leading publishers and content creators” for now, but Cloudflare says it will ensure “AI companies can use quality content the right way — with permission and compensation.”
Cloudflare has been helping domain owners fight AI crawlers for a while now. The company started letting websites block AI crawlers in 2023, but it only applied to ones that abide by a site’s robots.txt file, the unenforceable agreement that signals whether bots can scrape its content. Cloudflare began allowing websites to block “all” AI bots last year — whether they respect a site’s robots.txt file or not — and now this setting is enabled by default for new Cloudflare customers. (The company identifies scrapers to block by comparing them to its list of known AI bots.) Cloudflare also rolled out a feature in March that sends web-crawling bots into an “AI Labyrinth” to deter them from scraping sites without permission.
Several major publishers and online platforms, including The Associated Press, The Atlantic, Fortune, Stack Overflow, and Quora, are on board with Cloudflare’s new AI crawler restrictions, as websites contend with a future where more people are finding information through AI chatbots, rather than search engines. “People trust the AI more over the last six months, which means they’re not reading original content,” Cloudflare CEO Matthew Prince said during the Axios Live event last week.
Additionally, Cloudflare says it’s working with AI companies to help verify their crawlers and allow them to “clearly state their purpose,” such as whether they’re using the content for training, inference, or search. Website owners can then review this information and determine which crawlers to let in.
“Original content is what makes the Internet one of the greatest inventions in the last century, and we have to come together to protect it,” Prince said in the press release. “AI crawlers have been scraping content without limits. Our goal is to put the power back in the hands of creators, while still helping AI companies innovate.”
This time it's official: Pornhub is withdrawing from the state of Georgia. To help you continue watching your favorite content, we offer you a foolproof trick.
Tinder is trialing mandatory facial recognition security features in the US to verify profiles and crack down on impersonation and fake accounts. New users in California are now required to provide a biometric “Face Check” scan to confirm their face matches their profile photos for the dating service, Axios reported on Monday.
The Face Check feature involves taking a short video selfie that’s used to match biometric indicators and prove that the Tinder user isn’t a bot using artificially generated images, providing them with a verified badge upon completion. The scan will also check if the user’s face is being used in multiple accounts, which could help to prevent users from being impersonated or having their likeness used by deceptive “catfish” profiles.
Face Check is separate from Tinder’s ID Check feature, which uses government-issued ID to verify users’ age and identity, while Face Check seemingly only requires users to upload a selfie video. Tinder users have provided video selfies to verify their profiles since 2023, but verification wasn’t a mandatory requirement for creating a Tinder account. This change means that Californians will have to complete some version of verification if they want to use the platform at all.
“We see this as one part of a set of identity assurance options that are available to users,” Match Group’s head of trust and safety, Yoel Roth, told Axios. “Face Check … is really meant to be about confirming that this person is a real, live person and not a bot or a spoofed account.”
Tinder says the selfie video is deleted once verification is complete, but that the platform stores a “non-reversible, encrypted face map” to detect duplicate user accounts in the future.
The Face Check feature has already been piloted in Colombia and Canada, with Roth telling Axios that those tests showed “promising” results in “improving perceptions of authenticity” and reducing reports of bad actors. Tinder will now monitor how users in California respond to the Face Check feature before deciding if it should be rolled out more broadly across the US, according to Roth.
Huawei has open-sourced more of its AI models. Tech experts say it'll help the U.S.-blacklisted company continue to expand throughout the AI ecosystem.
