Reddit now blocks all search engines other than Google

Reddit can be a valuable source for information from real people, which is why Google is spending millions on a deal with the platform. But, now, Reddit has started to block many of its results from showing in other search engines properly.

In February of this year Google announced a new deal with Reddit which would see Reddit data used for training Google’s AI models as well as more prominently showing Reddit results within Google Search. Bloomberg reported that the deal was for around $60 million. In the time since, Reddit has also been showing in Google Search drastically more, often outranking websites that Reddit posts are linking to.

Now, Reddit results in other search engines are effectively being blocked.

This behavior was first reported by 404 Media, which notes that Reddit has updated its robots.txt file to block all bots from scraping any part of the site. In the file, Reddit says:

Reddit believes in an open internet, but not the misuse of public content.

In June, just prior to when the file was first updated, Reddit explained that the change was being made following “an uptick in obviously commercial entities who scrape Reddit” and “use Reddit content for any use case they want.” While it’s not said explicitly, it’s also pretty clear that AI training is a clear focus for this change in policy.

But, as a result, search engines other than Google are now not able to show proper results from Reddit as they previously did.

404 notes that Bing, DuckDuckGo, Mojeek, and Qwant are all affected, with results either not showing anything recent, or not showing the full site result. Kagi, a paid search engine, is apparently still showing data, but only because it buys some of its search index from Google, which continues to have access to Reddit data through the aforementioned deal.