For three decades, the internet ran on an unspoken deal: websites opened their doors to search engines for free, and in return, those engines sent readers their way. That arrangement is now falling apart. Artificial intelligence tools are crawling the web not to direct traffic, but to feed their own systems and generate answers—often without giving anything back.
The change matters because it threatens the economics of creating content online. When a website can’t count on search traffic to sustain it, the incentive to produce reliable, well-researched information shrinks. And the problem is already measurable.
Cloudflare, which manages infrastructure for roughly 30 percent of the internet’s top sites, estimates that more than half of all web traffic now comes from AI bots rather than human visitors. Some of these are supervised AI agents, but the bulk are automated crawlers that harvest content without ever sending a person to the source.
Related: Rising costs curb Australian dining out despite discounts
Site owners can ask crawlers to leave them alone through a standard file called robots.txt, but AI companies don’t always honor the request. When they do, a different problem emerges: sites pushing misinformation are far less likely to block crawlers, meaning AI answers end up drawing from lower-quality sources.
Wikipedia offers a window into what’s happening. Traffic to the English-language version of the site dropped sharply after Google rolled out AI summaries in that market. Similar patterns appeared in other languages as the feature expanded. Users increasingly get their answers without ever clicking through to the original source.
The phenomenon has a name in industry circles: “Google Zero.” That’s the point when direct traffic from Google dries up entirely for a given site. For some longtime web users, clicking through to verify an AI answer remains instinct. But that group is shrinking fast.
The Quality Problem Compounds
Many sites are now blocking AI crawlers outright. Starting September 15, Cloudflare began defaulting to block AI crawlers on pages containing advertising. That means up to 30 percent of the world’s top sites could vanish from AI Overviews summaries. It also means AI systems are increasingly training on text produced by other AI tools.
Related: SBI Funds listing falls short of Street expectations
This creates a feedback loop that’s hard to escape. Lower-quality content proliferates because it’s cheap to produce and doesn’t require the infrastructure that human-generated reporting needs. AI tools then ingest that content and produce outputs that reflect its mediocrity.
The web has weathered shifts before. The rise of social media changed how news spread, and paywalls changed how publications monetized content. Each transition created winners and losers, and the ecosystem adjusted in unexpected ways. What’s different this time is the speed and the directness of the threat to the underlying economics of information production.
For the moment, experts suggest a practical workaround: scroll past the AI summary and actually click on search results. It sounds simple, but it matters. Clicking sends traffic to content creators, which keeps them solvent. It also tends to deliver more context than a brief AI-generated answer can provide. The habit won’t solve the structural problems, but it helps while the industry figures out what comes next.
