In a recent analysis conducted by data journalist Ben Welsh, it was discovered that a significant number of news websites are blocking various AI bots. This has led to a divide among news publishers regarding their approach to these bots. This article will delve into the reasons behind this trend and the potential implications for the future of news outlets.
Analysis of the Situation
Welsh found that around 26% of the news websites surveyed are blocking Applebot-Extended, while 53% are blocking OpenAI’s bot. Google’s AI-specific bot, Google-Extended, is blocked by nearly 43% of the sites. This disparity in blocking rates suggests that Applebot-Extended may still be flying under the radar. However, Welsh noted that the number of sites blocking these bots has been gradually increasing over time.
There seems to be a divide among news publishers regarding whether to block AI bots or not. Some publishers may be withholding data until a partnership agreement is in place, while others may be concerned about copyright issues. For example, Condé Nast websites used to block OpenAI’s web crawlers but unblocked them after announcing a partnership with the company. Buzzfeed, on the other hand, blocks all AI web-crawling bots unless a partnership agreement is in place.
One of the major challenges faced by publishers is keeping an up-to-date block list, as new AI agents are constantly being introduced. This task is made even more difficult by the fact that robots.txt files need to be edited manually. As a result, some publishers are turning to services like Dark Visitors, which automatically update a client site’s robots.txt. This shows the growing importance of robots.txt in the digital publishing landscape.
It has been revealed that some CEOs from major media companies directly decide which AI bots to block. This level of involvement highlights the significance of this issue for media executives. Some outlets have explicitly stated that they block AI scraping tools because they do not have partnerships with their owners. This suggests that commercial agreements play a key role in determining which bots are allowed access to a website.
The blocking of AI bots by news websites is a complex issue that involves a range of factors, from copyright concerns to strategic partnerships. As the use of AI in the news industry continues to grow, it will be interesting to see how publishers navigate this evolving landscape. It is clear that the decisions made regarding which bots to block can have significant implications for the future of digital publishing.
Leave a Reply