Glossary

What is robots.txt?

robots.txt is a plain-text file at the root of a website that tells crawlers which paths they may or may not crawl, using User-agent, Allow and Disallow rules. It can also list your sitemap. It controls crawling, not indexing: a blocked page can still appear in results if other sites link to it.

By the AgentLaunch team · Updated November 2026 · Also called robots file, robots exclusion protocol

Syntax

User-agent: *
Disallow: /admin/

User-agent: GPTBot
Disallow: /

Sitemap: https://www.example.com/sitemap.xml
  • User-agent names the crawler (* means all).
  • Disallow blocks a path; Allow permits one inside a blocked path.
  • Sitemap points to your XML sitemap.
  • The most specific matching group for a crawler applies.

The Robots Exclusion Protocol dates back to 1994 and was standardised by the IETF as RFC 9309 in 2022.

Crawling vs indexing

Goal Use
Stop crawling a path robots.txt Disallow
Keep a page out of search results noindex meta tag or header (and allow crawling so it's seen)
Remove a page fast Search Console removals, then noindex

A common mistake is disallowing a page to remove it from Google: the crawler can't see the noindex, and the URL may still be indexed from links.

AI crawlers

robots.txt is where you set policy for AI crawlers: GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot, Google-Extended and others. Each company documents its tokens. Allowing AI search crawlers is important if you want to be cited in AI answers.

Common mistakes

  • Disallow: / left over from a staging site.
  • Blocking CSS and JavaScript Google needs to render pages.
  • Using robots.txt to hide private content (the file itself is public).
  • Forgetting that CDN bot rules can block crawlers robots.txt allows.

Example

Our own robots.txt allows search and AI crawlers, blocks one internal endpoint, and lists three sitemaps: the main site, the blog, and the directory and glossary pages.

A typical robots.txt for a SaaS

User-agent: *
Disallow: /app/
Disallow: /api/

User-agent: GPTBot
Disallow: /

Sitemap: https://www.example.com/sitemap.xml

This lets search engines and AI search crawlers read the public site, keeps them out of the logged-in app, opts out of one training crawler, and points to the sitemap. Adjust the AI groups to your own policy.

FAQ

Where does robots.txt go?

At the root of the host: https://www.example.com/robots.txt. Each subdomain needs its own.

Does robots.txt remove pages from Google?

No. Use noindex for that, and let Google crawl the page to see it.

Do all bots obey robots.txt?

Reputable crawlers do. Malicious bots ignore it.

Can I block AI crawlers but not Google?

Yes. Add groups for each AI user agent; Googlebot follows its own group.

How do I test my robots.txt?

Search Console has a robots.txt report; fetch the file yourself and check each crawler's group.

Sources

  1. Google Search Central: Introduction to robots.txt
  2. RFC 9309: Robots Exclusion Protocol
  3. OpenAI: Overview of OpenAI crawlers

Data

All glossary terms · Search and AI search terms