Glossary
What is GPTBot?
GPTBot is OpenAI's web crawler that collects publicly available pages that may be used to train its AI models. It identifies itself with the user agent token GPTBot and respects robots.txt. Blocking GPTBot does not remove a site from ChatGPT search, which uses a separate crawler, OAI-SearchBot.
OpenAI's three bots
| Bot | Purpose |
|---|---|
| GPTBot | Crawl content that may be used for training |
| OAI-SearchBot | Find and surface sites in ChatGPT search results |
| ChatGPT-User | Fetch pages when a ChatGPT user or custom GPT asks |
OpenAI documents that each setting is independent: a site can block GPTBot and still allow OAI-SearchBot so it appears in search answers.
Blocking it
User-agent: GPTBot
Disallow: /
To allow it, either don't mention it or add Allow: /. Changes take effect as OpenAI rereads your robots.txt. OpenAI publishes the IP ranges its bots use so you can verify traffic.
Should you block it?
That's a content policy choice. Publishers protecting paid content often block training crawlers. Products that want to be known by future models may allow it, since training data is one way a model learns that a product exists. Blocking it has no effect on ChatGPT search citations.
Example
A documentation site allows GPTBot and OAI-SearchBot. A news site blocks GPTBot but allows OAI-SearchBot to keep getting cited. A startup blocked both by mistake through a CDN "block AI bots" setting, and fixed it after noticing zero ChatGPT referrals.
FAQ
Does GPTBot obey robots.txt?
OpenAI says it does.
Will blocking GPTBot remove me from ChatGPT?
Not from ChatGPT search, which uses OAI-SearchBot. It affects training data.
How do I verify GPTBot traffic?
Check the source IPs against the ranges OpenAI publishes.
Sources
Data
- llms.txt and AI crawler statistics 2026: 508 sites crawled
- AI search statistics 2026: AI Overviews, ChatGPT and citations