OPEN OR CLOSED

Right now, publishers are making one of the most important decisions about their future — without realising they're making it.

It's not happening in strategy meetings. Not in boardrooms. It's happening in four small files that sit on every publisher's website — robots.txt, ai.txt, llms.txt, JSON-LD. They tell AI systems whether they can come in, who you are, and how to cite you. Most publishers have never opened them. Some don't know they exist.

I've worked with publishers for a long time. Like most people here, I've watched the traffic change. It didn't disappear — people didn't stop being curious about the world. It went somewhere else. People started asking AI assistants and getting answers directly, without ever arriving at a publisher's site.

I wanted to understand what that means in practice. How do you become the answer? What does it take? What are publishers around the world actually doing about this right now? And — given the EU AI Act's transparency provisions, even after the recent delay — is anyone ready?

So I spent four months on the data. Close to 5,000 publisher domains across 99 countries. AI policies, technical configurations, whether the doors are open or closed, and whether any of it looks like a deliberate choice.

What I found was interesting. And a little uncomfortable.

—

For a long time, the game was clear. You Googled. If you were on page one as a publisher, you were in good shape. It was hard to get there, but you could understand it. You could watch your analytics. You could see where you stood.

There is no page one anymore. And most publishers are still optimising for it.

There is just the answer. When someone asks an AI assistant about your coverage area — your city, your beat, the topic you've spent years building expertise on — you don't know if you're mentioned. You get no notification. No analytics. Either you're in the answer or you're not.

The rules exist. They're just invisible to most people working in publishing.

We've seen this shape before. Kodak invented digital photography and then spent two decades protecting the film business that was paying the bills. Fujifilm read the same room and built something new while the old model still had margin. One survived. One didn't. It wasn't that Kodak was worse at photography. It's that they treated the new distribution layer as a threat to manage instead of a system to enter.

With music, books, restaurants — the pattern repeats. A platform emerges that's genuinely better for the audience: easier, faster, everything in one place. And the creators who built their value on a direct relationship with their audience find that relationship has quietly moved somewhere else. The platform didn't win because it was better than any individual creator. It won because it removed the need to choose.

That's what's happening again. The difference is speed. With music it took a decade. With AI assistants, the behaviour shift is already here. The risk isn't that you slowly fade over fifteen years. It's that on your core beats, AI answers replace you in a fraction of that.

—

What the data shows is that most publishers aren't actively using these tools. Or if they are, it looks more like an inherited configuration than a decision. For the technical detail on each of these files, see What the Bot.

Of close to 5,000 publishers analysed across 99 countries: 21% block all AI crawlers. 4% block some. 47% allow all. And 28% have set no policy at all — more than one in four publishers hasn't made a choice. Not open. Not closed. Just absent.

The country picture is sharper. Finland and Norway block at 67%, Sweden at 66% — three times the global average. In the United States, roughly half of major publishers block all AI crawlers.

These numbers look like decisions. Mostly they're default behaviour. Configurations set years ago, before anyone was thinking about AI. CMS settings that block bots as a general rule. Choices inherited, not made.

Most publishers have an AI policy. They just didn't choose it.

—

So — open or closed? Here's what I think after four months in the data: it's the wrong question.

Close the door and you protect your content from being used in future training. That matters. But blocking doesn't reach back to what's already in the model, and it doesn't shape how the AI represents you today. When someone asks about your beat, the AI reaches for whoever is available. If it isn't you, it's an aggregator, a content farm, or Reddit. By opting out, you're not just protecting your IP. You're removing yourself from the primary discovery engine of the next generation.

Blocking protects your content from being used in training. It does not protect your authority from being replaced.

Open the door and you may lose the visit. The reader gets the answer. Your counter doesn't move. For publishers heavily reliant on ad-funded pageviews, you're providing the data that trains the tool that replaces you. And being open isn't enough on its own. An AI system that can crawl your site but finds no structured data, no authorship signals, no sense of who you are — will represent you badly, or not at all.

You can be open and still be invisible.

—

The real question isn't open or closed. It's whether your current position is a choice or a default. In my research across 5,000 domains, for most publishers it's a default — a technical setting left over from a different era. Blocking without a formal declaration is a technical instruction, not an institutional record. Letting AI systems in without telling them who you are is presence without identity.

The publishers who'll navigate this well won't be the ones who got the binary right. They'll be the ones who decided deliberately, understood the cost, and built what it takes to be found, understood, and cited correctly.

Kodak had the technology. They didn't have the decision.

The window is still open. The pattern says it doesn't stay open long.

Default is not neutral. It is a position — with consequences.

—