AI crawler access: can AI bots read your site?

Every AI answer starts with a fetch. Before ChatGPT, Gemini, Perplexity, or Copilot can name your business, one of their crawlers has to visit your page and read it. If that fetch is blocked, you are invisible to AI no matter how good your website is. This is the first thing our free audit checks, because it is the one problem that makes every other fix pointless.

And more of your customers are asking AI first. BrightLocal's consumer research found the share of people using AI to find local businesses jumped from about 6 percent to about 45 percent in a single year. The businesses that AI can read are the ones getting named. This page is the plain-English guide to whether AI can read yours, and how to fix it if it cannot.

What is an AI crawler?

An AI crawler is a bot that an AI company sends to fetch and read web pages. It works like a search-engine crawler, but the pages it collects feed an AI system instead of a classic search index. Some AI crawlers gather text to train a model. Others fetch a page live, in the moment, to help answer a specific question and cite a source.

That difference matters more than anything else on this page, so hold onto it. A training crawler and a live-answer crawler are not the same bot, they do not use the same name, and blocking one does not block the other.

Which AI crawlers should you know?

Here are the AI crawlers that matter in 2026, what each one feeds, and the exact name to use in your robots.txt file. The "verify" column links to the official list each company publishes, so you can confirm a bot is real and not an impostor using its name.

One honest note on impostors. A user-agent name is just text, so anyone can send a bot claiming to be GPTBot. The real crawlers publish their IP address ranges in the JSON files above, and Googlebot and Bingbot can be confirmed by a reverse DNS lookup. That is how you tell a real AI crawler from a scraper wearing its name.

What is GPTBot?

GPTBot is OpenAI's crawler for collecting web content that may be used to train its models. It reads your pages, and OpenAI honors the rules you set for it in robots.txt using the name GPTBot. OpenAI publishes GPTBot's IP ranges at openai.com/gptbot.json so you can confirm a visit is genuine.

GPTBot is often confused with ChatGPT's live search, and that confusion leads to a costly mistake. GPTBot is the training bot. When a person asks ChatGPT a question and it browses the web to answer, that is a different bot, either OAI-SearchBot or ChatGPT-User. So blocking GPTBot stops OpenAI from training on your content, but it does not remove you from ChatGPT's live, cited answers. Those depend on the other two names. Most "should I block GPTBot" advice online misses this completely.

How do I check if AI crawlers can access my site?

You check in three places, from easiest to deepest. Most blocks hide in the second one.

1. Your robots.txt file. Open yourdomain.com/robots.txt in a browser. Look for any Disallow lines aimed at the crawler names in the table above. A line like User-agent: GPTBot followed by Disallow: / means that bot is fully blocked.
2. Your CDN or security layer. This is where the silent blocks live. In Cloudflare, check the bot and AI-crawler controls, because there is a one-click setting that blocks AI bots and it can be on without anyone remembering to turn it off. Other hosts and firewalls have their own version of this.
3. Your server logs. Search your access logs for the crawler names. If you see GPTBot, PerplexityBot, and the others fetching pages and getting a 200 response, they are getting in. A wall of 403 responses means they are being turned away.

If that sounds like a lot to check across every engine and every page, that is exactly what our free AI-Visibility Audit does for you. It fetches your site the way the AI crawlers do and tells you, in plain terms, which ones can read you and which ones are blocked.

How do I block or allow AI crawlers?

You control access with robots.txt for the well-behaved bots, and with your CDN for everything else. Here is the plain version.

To welcome AI crawlers (what most businesses want)

Do nothing that blocks them, and make sure your CDN is not blocking them either. A clean, open robots.txt that points to your sitemap is enough. It looks like this.

To block one specific AI crawler

Name the bot and disallow it. For example, to stop OpenAI from training on your content while still allowing ChatGPT's live search to cite you, block only GPTBot.

Two cautions. First, robots.txt is a public request, not a lock. The named bots honor it, but it will not stop a bad-faith scraper. Second, some user-triggered fetchers do not follow robots.txt at all. Perplexity states that Perplexity-User generally ignores robots.txt, because a person asked for that page directly. To hard-block a bot regardless, you use your CDN, not robots.txt.

Should I block AI crawlers?

For a real business, no. Blocking AI crawlers hands your AI visibility to your competitors, and there is almost no upside to weigh against that. This is the claim, and here is the case for it.

Why it matters. If a live-answer crawler cannot fetch your pages, you are not eligible to be cited, full stop. As one plain way to put it, an AI cannot cite what it cannot read. Every other tactic, better content, reviews, schema, all of it, assumes the bot got in first. Blocking the crawler removes you from the game before it starts.

The proof, and the honest counter-case. The only people with a real reason to block these bots are those hiding content they do not want associated with them, such as operators of disposable, mass-produced AI-spam sites who block GPTBot to delay detection. Even in that world it only delays detection, it does not prevent it. There is a narrow, legitimate reason a publisher might block a training crawler, which is to protect original work from being used to train a model for free. That is a copyright and licensing choice, and it is fair. But notice it is a decision about training, using the training-bot names, and it is separate from live citation. If your goal is to be recommended by AI, blocking the crawlers is the one move that guarantees you will not be.

Does allowing crawlers get you cited?

No, and this is where honest and dishonest providers split. Letting the crawlers in is necessary, but it is not what wins the citation. Access is the floor you stand on, not the prize.

The evidence is clear that ranking and citation are two different games. Ahrefs studied 15,000 queries and found only about 12 percent of the URLs cited by AI also rank in Google's top 10 (Ahrefs sells SEO tools, so read it with that in mind, but the finding cuts against their own product). In other words, most AI citations go to pages that are not ranking at all. What they have in common is that a crawler could read them, and the content earned the mention.

So access gets you into the room. Getting chosen inside that room is won mostly off your own site, through clear, quotable content and, above all, other trusted places mentioning your business. That is the real work of generative engine optimization. If you want the deeper version of why a good business still stays invisible, read why isn't my business showing up in ChatGPT. And two things people expect to be levers are not: read our honest take on whether llms.txt works and on schema markup for AI search.

Common questions about AI crawler access