As the US government weighs its options following a landmark ruling against Google last week, online publications are increasingly looking at a bleak future. (And this time, it’s not just because of sharply reduced advertising revenue.) Bloomberg reports that they now have the choice of allowing Google to use their published content to produce inline AI-generated search answers or lose their visibility in the company’s search engine.
The crux of the problem lies with Googlebot, the crawler that crawls and indexes the live web to produce the results you see when you enter search terms. If publishers prevent Google from using their content for the AI-generated answers you now see at the top of many search results, they also lose the privilege of having their web pages included in standard web results.
The dilemma has led publications, rival search engines and AI startups to pin their hopes on the Justice Department. On Tuesday The New York Times reported that the Justice Department is considering asking a federal judge to break up parts of the company (spin off divisions like Chrome or Android). Other options it is reportedly considering include forcing Google to share search data with competitors or abandoning its standard search engine contracts, like the $18 billion deal with Apple.
Google uses a separate crawler for its chatbot Gemini (formerly Bard), but the main crawler covers both AI summaries and standard searches, so web publishers have little (if any) influence. If you let Google crawl your content for answers to AI summaries, readers may consider the matter settled without bothering to visit your site (which means those potential readers will not earn any revenue). But if you block Googlebot, you will lose visibility in search, which will likely mean significantly less short-term revenue and a huge loss of long-term competitive position.
iFixit CEO Kyle Wiens said Bloomberg“I can block ClaudeBot (Anthropic’s crawler for the chatbot Claude) from indexing us without hurting our business. But if I block Googlebot, we lose traffic and customers.”
Another problem with combining the two is that it gives Google an invaluable advantage over smaller AI startups. The company gets a wealth of free training data from publishers who want to remain visible in search results. In contrast, AI companies are forced to pay publishers for access to their data – and even then, that wouldn’t be worth as much as the huge advantage Google gets (essentially) for free.
From this perspective, it is not surprising to read that according to BloombergGoogle turns away publishers who try to negotiate content deals (Reddit was the only exception). Why waste money on content deals when you can get all the training data you want in exchange for the search results that most publishers need to survive?
“Now there are a lot of technology companies that are paying for content, they’re paying for access to it because they need it to seriously compete,” said Alex Rosenberg, CEO of AI startup Tako Inc. Bloomberg“Google, on the other hand, doesn’t actually have to do that.”
It all comes down to the leverage that Google has over desperate publishers. In addition to the industry’s existing financial problems (online advertising revenues have plummeted over the past eight years), AdWeek reported in March that Google’s AI-generated search results could lead to a 20 to 60 percent drop in organic search traffic.
The ball is now in the Justice Department’s court to decide what happens next with Google – and to some extent with the entire Internet. BloombergThe whole story is worth reading.
