Reddit has up to date its robots.txt file, stopping Bing and plenty of different serps from crawling the location. “Bing stopped crawling Reddit after they implemented their updated robots.txt file on July 1, which prohibits all crawling of their site,” a Microsoft consultant advised Search Engine Land.
What occurred. On July 1, 2024, Reddit up to date its robots.txt file stopping many serps and AI instruments from crawling the location. Reddit didn’t stop Google from crawling the location, regardless of what some individuals could have thought earlier this month. However Reddit did stop most different crawlers from crawling the location.
Earlier this morning, Mark Williams-Prepare dinner notified me that Reddit outcomes had been dropping out of the Bing Search index. Then a number of media shops started masking the information. I needed to substantiate that Bing’s crawlers had been certainly blocked, as a result of Reddit was utilizing IP detection to point out serps one model of its robots.txt file and people one other model of the robots.txt file – as I defined earlier this month.
Bing has thus stopped crawling new content material on Reddit, which is why once you filter Reddit ends in Bing Seek for the final week, you see nothing:
Microsoft confirmed. A Microsoft spokesperson advised Search Engine Land:
“We respect the robots.txt standard. Bing stopped crawling Reddit after they implemented their updated robots.txt file on July 1, which prohibits all crawling of their site.”
Reddit assertion. Reddit spokesperson Tim Rathschmidt says in a press release to The Verge:
“This is not at all related to our recent partnership with Google. We have been in discussions with multiple search engines. We have been unable to reach agreements with all of them, since some are unable or unwilling to make enforceable promises regarding their use of Reddit content, including their use for AI.”
Why we care. With Reddit securing a licensing take care of Google, Reddit is ready to play hardball with different serps and AI instruments. So Reddit has blocked most different serps from crawling its content material. In the meantime, Google is driving insane visitors to Reddit nowadays, together with testing particular remedy in its search outcomes for them.
It makes you marvel if different giant web sites can attempt to go down this route and the place which may depart smaller publishers and content material producers.
In the meantime, don’t count on to see a lot new Reddit content material being surfaced on Bing within the close to future.
