I read something this week that made me rethink how we measure AI visibility. It wasn’t about content. It was about infrastructure.
Cloudflare has just told AI companies they’ve got until September 15th to split their crawlers into two clear buckets: search, or training and agent use. Miss that deadline, and ad-supported publisher sites will block the mixed-use crawlers by default.
At the same time, Cloudflare’s Pay Per Crawl is becoming Pay Per Use, meaning publishers get paid when their content actually shows up in an AI answer, not just when a bot grabs the page.
On the surface, that’s a story about publisher economics. Underneath, it’s a story about who gets cited in AI search, and it has very little to do with how good your content is.
Why this matters for GEO
Most GEO advice focuses on the content layer: structure it to be cited, use quotes and stats, keep formatting clean. All of that is genuinely useful, and none of it is wrong.
But this is a reminder that citation is increasingly decided one level down, at the infrastructure layer, by who has a crawler deal with who. Not by how quotable your content is.
You can land the perfect placement in a top tier outlet and still be invisible in ChatGPT, if that outlet blocks the crawler or isn’t part of a paid deal. Coverage and AI citation are no longer the same win.
What this changes about a media list
A media list used to answer one question: will this outlet publish us? After September, it needs to answer two.
- Does the target publication actually allow AI crawlers? Most publish this right in their robots.txt, and it’s a five-minute check per outlet.
- Are earned coverage and AI citation visibility being reported as separate lines? After September they won’t move together the way people assume. A great placement in a blocked outlet earns brand credibility but nothing in AI answers.
If a client publishes real content volume of their own – blogs, guides, research – their own crawler settings also need flagging before the deadline too. It’s not just a media relations question, it’s a technical one.
Two different visibility games are about to run side by side: traditional earned coverage, and AI citation visibility. They used to move together, but rom September, they won’t.
Which tier 1 publications actually allow AI crawlers right now
We pulled together the current picture across top UK and US titles, based on a robots.txt analysis of the top 50 news sites in each market.
It splits into three groups:
- Outlets that are fully open to AI crawlers
- Outlets that selectively allow the crawlers that actually drive citations (OAI-SearchBot, ChatGPT-User, Claude-Web, Perplexity-User) while blocking pure training bots
- Outlets that block more or less everything.
See the table image for the full breakdown – data from Buzzstream
The takeaway
Citation is no longer purely an earned media problem or purely a content problem. It’s becoming an infrastructure and licensing problem, decided by crawler agreements most comms and marketing teams have never looked at.
The agencies and in-house teams who start treating AI crawler access as a standard part of media list building, right alongside domain authority and audience fit, are the ones who’ll be able to explain to clients why a brilliant piece of coverage did or didn’t show up in ChatGPT.