AEO

Robots.txt for AI Crawlers: GPTBot, ClaudeBot, Google-Extended

By · · 2 min read

Short answerAI companies run separate crawlers for training, search and user-requested visits, and robots.txt controls them individually. Blocking a training bot such as GPTBot or ClaudeBot is not the same as blocking a search bot such as OAI-SearchBot or Claude-SearchBot, which can reduce how often your site appears in AI answers.

Key takeaways

  • OpenAI, Anthropic and Google each separate training crawlers from search or user-initiated access, so decide per bot rather than all at once.
  • Blocking OAI-SearchBot or Claude-SearchBot can reduce visibility in AI search answers; blocking GPTBot or ClaudeBot targets model training.
  • Google says Google-Extended does not affect inclusion in Google Search or act as a ranking signal.
  • OpenAI says robots.txt changes take about 24 hours to apply, and user-initiated fetches such as ChatGPT-User may not follow robots.txt rules.

Why there is more than one AI crawler

AI companies no longer run a single bot. Each of the three largest players separates crawlers by purpose, so a robots.txt file can allow one activity and refuse another. The purposes are model training, search indexing and user-requested page visits. Treating them as one group is the most common mistake.

What OpenAI's crawlers do

According to OpenAI's crawler documentation, OAI-SearchBot powers ChatGPT's search features and surfaces websites in search results. Disallowing it prevents a site from appearing in ChatGPT search answers, though the site may still appear as a navigation link. GPTBot crawls content for training generative AI foundation models, and disallowing it signals that content should not be used for training. ChatGPT-User handles actions that a user starts inside ChatGPT, and OpenAI notes that because a person initiates these visits, robots.txt rules may not apply. OpenAI also says changes to robots.txt take approximately 24 hours to take effect.

What Anthropic's crawlers do

Anthropic's help documentation describes three bots. ClaudeBot collects web content that could contribute to model training; blocking it excludes future material from training datasets. Claude-User supports people asking Claude questions and may fetch a site on their behalf; blocking it prevents retrieval for those requests and can reduce visibility. Claude-SearchBot navigates the web to improve search result quality; blocking it prevents indexing for search and can reduce visibility in results. Anthropic says it honors industry-standard robots.txt directives and that the rules apply per subdomain.

What Google-Extended is, and is not

Google describes Google-Extended as a standalone product token that publishers can use to manage whether content Google crawls may be used for training future generations of Gemini models and for grounding in Gemini applications. Google states that Google-Extended does not impact a site's inclusion in Google Search and is not used as a ranking signal. It also has no separate HTTP user agent string: crawling uses existing Google user agents, and the token acts only as a control in robots.txt. That means blocking it is a training and grounding decision, not a search decision.

How to decide

A short example

A site that wants to be findable in AI search but opt out of training could allow OAI-SearchBot and Claude-SearchBot while adding a Disallow rule for GPTBot and ClaudeBot. Because OpenAI says changes take about a day to apply, check the result the following day rather than immediately.

Limits to keep in mind

Robots.txt is a voluntary standard. It governs crawlers that choose to honor it, and the documentation notes that user-initiated fetches may not follow it. It also says nothing about content already collected. Review each provider's current documentation periodically, because crawler names and behavior change.

Frequently asked questions

Does blocking GPTBot remove my site from ChatGPT search?

According to OpenAI, GPTBot is used for training and OAI-SearchBot powers ChatGPT search, so they are controlled separately. Disallowing GPTBot signals that content should not be used for training.

Will blocking Google-Extended hurt my Google rankings?

Google states that Google-Extended does not impact a site's inclusion in Google Search and is not used as a ranking signal.

How long does a robots.txt change take to work?

OpenAI says robots.txt updates take approximately 24 hours to take effect in its systems. Other providers may differ.

Sources

  1. Overview of OpenAI crawlers — OpenAI
  2. Does Anthropic crawl data from the web, and how can site owners block the crawler? — Anthropic
  3. Google's common crawlers — Google Search Central
KC
Kathy P. Carr

CEO of AIrecommend.ai and Skyview Business Consulting. Writes on answer engine optimization, franchising and building AI-first companies.

Connect on LinkedIn

Keep reading

SEOCan You Earn a Google Featured Snippet? What Google SaysGoogle says you can't mark up a page to earn a featured snippet. Here is what its documentation says about how snippets are chosen, and how to control them.SEODoes Google Penalize Duplicate Content?Google's docs frame duplicate content as a version-selection issue, not a penalty. Learn what causes duplicates, when it becomes spam, and how to fix it.SEONoindex vs Robots.txt: Keeping Pages Out of GoogleRobots.txt controls crawling, not indexing. Learn when to use noindex, why blocking and noindex together fail, and when passwords are the right tool.
Want to be the answer AI recommends?
See how AIrecommend.ai helps businesses get found and recommended by AI assistants.