Reddit will proceed blocking Microsoft and different search engines like google and yahoo and AI fashions from crawling its content material utilizing robots.txt – until they strike a content material licensing deal. That’s what Reddit CEO Steve Huffman mentioned in a brand new interview.
Why Reddit is obstructing search engines like google and yahoo. Noting it’s been “a real pain in the ass to block these companies,” right here’s what else Huffman instructed expertise information website The Verge:
- “Without these agreements, we don’t have any say or knowledge of how our data is displayed and what it’s used for, which has put us in a position now of blocking folks who haven’t been willing to come to terms with how we’d like our data to be used or not used.”
- “We’ve had Microsoft, Anthropic, and Perplexity act as though all of the content on the internet is free for them to use. That’s their real position.”
- “I think the traditional value exchange from search engines has changed. Search and summarization and training are merging, and the value exchange of crawling in exchange for traffic back is becoming muddied.”
Freeware. Sure, Microsoft AI CEO Mustafa Suleyman really referred to as net content material “freeware,” saying anyone can copy and use it:
- “…With respect to content that is already on the open web, the social contract of that content since the ’90s has been that it is fair use. Anyone can copy it, recreate with it, reproduce with it. That has been freeware, if you like. That’s been the understanding.”
Google not blocked. In the meantime, Reddit didn’t block Google. That’s as a result of Google pays Reddit $60 million a 12 months. That content material licensing deal was introduced in February.
Microsoft assertion. Following the information about Reddit blocking search engines like google and yahoo, a Microsoft spokesperson instructed Search Engine Land:
- “Microsoft respects the robots.txt customary and we honor the instructions supplied by web sites that are not looking for content material on their pages for use with our generative AI fashions. Bing stopped crawling Reddit after they carried out their up to date robots.txt file on July 1, which prohibits all crawling of their website.“
Why we care. Reddit is in a strong place, having a licensing cope with Google – to not point out the insane quantity of natural visibility and visitors it’s getting because of its prominence in Google Search outcomes. Nonetheless, different content material producers and publishers probably will want no matter visibility and visitors they’ll get from AI search and reply engines by incorporating generative engine optimization (GEO) methods.
New on Search Engine Land
Concerning the writer
Danny Goodwin is Editorial Director of Search Engine Land & Search Advertising Expo – SMX. He joined Search Engine Land in 2022 as Senior Editor. Along with reporting on the most recent search advertising and marketing information, he manages Search Engine Land’s SME (Topic Matter Knowledgeable) program. He additionally helps program U.S. SMX occasions.
Goodwin has been modifying and writing in regards to the newest developments and tendencies in search and digital advertising and marketing since 2007. He beforehand was Government Editor of Search Engine Journal (from 2017 to 2022), managing editor of Momentology (from 2014-2016) and editor of Search Engine Watch (from 2007 to 2014). He has spoken at many main search conferences and digital occasions, and has been sourced for his experience by a variety of publications and podcasts.
