Are you accidentally blocking AI crawlers? How to check your robots.txt

Yes, you might be. Many websites block the crawlers of AI assistants without ever having decided to: a rule inherited from a template, a security plugin or a setting from the hosting provider. If an AI search tool can’t read your page, there is no content to work on: it’s the first thing we check in any GEO project, and it can be done in an afternoon.

Why can a robots.txt block keep you out of AI answers?

The robots.txt file tells crawlers which parts of your website they may visit. AI assistants with search need to read pages in order to use them as sources. If you’ve told them no, your content is out of the game before anyone has judged whether it’s any good.

A caveat: letting a crawler in doesn’t mean you’ll be cited. It is a necessary condition, not a guarantee. What decides things afterwards is the quality, clarity and accuracy of what you’ve published, and no technical file controls that.

In Google’s case, its documentation on AI features in Search says that for a page to appear as a supporting link in AI overviews, it must be indexed and eligible to be shown with a snippet in results. It asks for no additional technical requirements. You can read it in Google’s official documentation. If your site already works well for traditional SEO, you have that base covered.

Which crawlers should you tell apart?

Not all AI bots do the same thing. Some collect content to train models; others read it to answer a specific query. Mixing them up leads to blocking too much or too little. OpenAI, for example, separates its crawlers and explains that they are controlled independently in robots.txt, according to its bots documentation:

• OAI-SearchBot: used to show websites in ChatGPT search results. If you block it, you’re removing your site from that search.
• GPTBot: associated with training OpenAI’s models. Blocking it is a decision about whether your content is used for training, separate from the previous one.
• ChatGPT-User: acts when a user asks the assistant to visit a specific page.
• Googlebot and Google-Extended: Googlebot is the Search crawler. Google-Extended is a robots.txt signal for the use of content in its generative AI models. According to Google’s crawlers documentation, it does not affect your inclusion in Search.
• PerplexityBot: Perplexity’s crawler, which publishes its own documentation. It’s worth reading it before you decide.

The names and functions of these bots change over time. Always check each provider’s documentation at the moment you edit the file, not a list from months ago.

How do I check my website step by step?

1. Open your robots.txt by typing yourdomain.com/robots.txt in the browser. Look for rules such as “Disallow: /” under a specific user-agent or under “*”.
2. Note which bots are blocked and who put the rule there. If nobody on your team knows, ask whoever manages the website or the hosting.
3. Check other layers: some security services, CDNs or plugins block automated bots outside robots.txt. A correct file doesn’t guarantee that the crawler actually reaches the page.
4. Check the noindex tag and snippet limits on your important pages. If a page isn’t indexed or doesn’t allow snippets, Google doesn’t consider it for its AI features.
5. Decide by type of content (see the next section) and leave the rule written down, with a date and the reason.
6. Check again after a while: changes to robots.txt are not reflected instantly by crawlers.

Should I let in all AI bots?

Not necessarily, and it’s a business decision, not a technical one. There are two different questions that tend to get mixed up:

• Do I want my content to be able to serve as a source when someone asks about my sector? If your business depends on being found (services, enquiries, products), it usually makes sense to let search crawlers in.
• Do I want it to be used to train models? That is a question of content ownership. Some companies prefer to exclude it, and that is legitimate.

You can also be selective: open the blog and the service pages and close off private areas, duplicate listings or paid content. What you shouldn’t do is block everything by default without knowing what you lose.

And after robots.txt, what next?

Once access is sorted, the real work begins. At Altoclick we review whether your pages clearly answer your customers’ real questions, whether the data (prices, location, terms, services) is accurate and up to date, and which external sources talk about you. Then we measure what the assistants say about your company: we record mentions, citations, accuracy and sources, with date and engine, so we can compare over time without relying on a single screenshot. We don’t promise you’ll appear in any answer: we work on what does depend on you, which is useful, well-supported content.

If you want to see how this fits with the rest of your online presence, you’ll find more context on our online visibility page and in the GEO service. And if you’d rather start with a diagnosis, you can request the free GEO audit: it includes a review of crawler access, and we tell you what is worth changing and what isn’t.