As media organizations negotiate licensing agreements with AI giants like OpenAI in search of training data, they are simultaneously implementing barriers to block web crawlers used by AI entities to gather data. Recent data indicates that more than 88 percent of the leading news outlets in the US now restrict web crawlers utilized by AI companies for chatbot and other AI initiatives. Interestingly, right-wing media outlets notably lag behind their liberal counterparts in implementing bot-blocking measures.
Research conducted by Ontario-based AI detection startup Originality AI in mid-January on around 40 prominent news websites revealed that nearly all of them inhibit AI web crawlers. This includes well-known newspapers such as The New York Times, The Washington Post, and The Guardian, as well as magazines like The Atlantic and niche sites like Bleacher Report. The most commonly blocked crawler is OpenAI’s GPTBot. However, among the top right-wing news sites examined, including Fox News, the Daily Caller, and Breitbart, none of them impede the most prominent AI web scrapers, which also encompass Google’s AI data collection bot. Notably, The Free Press, a new website by pundit Bari Weiss, also does not obstruct AI scraping bots.
While most right-wing sites did not provide feedback on their AI crawler strategy, experts speculate on the reasons behind this discrepancy. One hypothesis suggests that this approach may be a tactic to counter perceived political bias in AI models. The concern stems from the fact that AI models tend to reflect the biases present in their training data. By allowing access to right-leaning content while left-leaning sites block access, these outlets may be attempting to influence the balance of perspectives in AI algorithms.
Originality AI determined the sites that block GPTBot and other AI scrapers by examining the robots.txt files that websites use to communicate with automated web crawlers. The analysis focused on top US news sites based on estimated web traffic, with particular attention to right-leaning outlets due to their scarcity in the sample. Despite concerns about bias in AI tools, the impact of right-wing sites allowing AI scraping remains debatable, given the extensive data already collected from mainstream sources and the neutral stance pursued by most AI companies.
The disparity in AI crawler blocking practices could also reflect ideological differences regarding copyright issues. While mainstream media like The New York Times have taken legal action against AI entities for copyright infringement, right-wing media’s stance has been less vocal. Some right-leaning outlets cited oversight or lack of awareness as reasons for permitting AI scrapers, prompting them to reconsider their approach.
In conclusion, the debate surrounding AI crawler blocking reveals a complex interplay of technical, ideological, and legal considerations within the media landscape. As media outlets navigate the evolving AI ecosystem, decisions regarding data access and copyright protection will continue to shape the dynamics between traditional and emerging digital platforms.
