llms.txt and AI crawler statistics 2026: 508 sites crawled

llms.txt adoption

Source: AgentLaunch crawl of robots.txt, llms.txt and homepages, October 2026.

• 115 of 196 software and AI companies (59%) publish an llms.txt file.
• 146 of 312 startup directories (47%) publish one.
• 25 companies (13%) and 48 directories (15%) also publish llms-full.txt.
• 8% of sites with llms.txt still block at least one AI training crawler.
• AI tool directories: 34% publish llms.txt (110 crawled).
• Launch platforms: 64% publish llms.txt (90 crawled).
• SaaS and startup directories: 56% publish llms.txt (34 crawled).
• Software review sites: 35% publish llms.txt (26 crawled).
• Developer tool directories: 42% publish llms.txt (12 crawled).
• Alternatives directories: 30% publish llms.txt (10 crawled).
• Startup databases: 11% publish llms.txt (9 crawled).
• MCP registries: 100% publish llms.txt (6 crawled).
• AI agent directories: 63% publish llms.txt (8 crawled).
• At least one launch platform's llms.txt is written as instructions for AI agents submitting startups on a user's behalf.

AI crawlers blocked by startup directories

Share of 312 live directories whose robots.txt fully blocks each bot (Disallow: / for that bot or for all bots).

Source: AgentLaunch crawl of robots.txt, llms.txt and homepages, October 2026.

• GPTBot: blocked by 9% of directories, named in 26% of robots.txt files.
• OAI-SearchBot: blocked by 1% of directories, named in 15% of robots.txt files.
• ChatGPT-User: blocked by 1% of directories, named in 16% of robots.txt files.
• ClaudeBot: blocked by 8% of directories, named in 25% of robots.txt files.
• Claude-SearchBot: blocked by 1% of directories, named in 8% of robots.txt files.
• anthropic-ai: blocked by 9% of directories, named in 20% of robots.txt files.
• PerplexityBot: blocked by 1% of directories, named in 20% of robots.txt files.
• Google-Extended: blocked by 8% of directories, named in 22% of robots.txt files.
• CCBot: blocked by 11% of directories, named in 20% of robots.txt files.
• Bytespider: blocked by 14% of directories, named in 18% of robots.txt files.
• Applebot-Extended: blocked by 8% of directories, named in 20% of robots.txt files.
• meta-externalagent: blocked by 10% of directories, named in 16% of robots.txt files.
• Amazonbot: blocked by 11% of directories, named in 15% of robots.txt files.
• 95% of startup directories (295 of 312) have a robots.txt file.
• 88% list a sitemap in robots.txt.
• 47% publish an llms.txt file (146 sites).
• 15% also publish llms-full.txt.
• 29% name at least one AI crawler in robots.txt.
• 12% fully block at least one major AI training crawler (GPTBot, ClaudeBot, CCBot or Google-Extended).
• 8% block GPTBot (training) while leaving OAI-SearchBot (ChatGPT search) allowed.
• 85% block none of the 13 AI crawlers we checked.

AI crawlers blocked by software and AI companies

The same checks for 196 well-known software, developer tool and AI companies.

Source: AgentLaunch crawl of robots.txt, llms.txt and homepages, October 2026.

• GPTBot: blocked by 3% of companies, named in 17%.
• OAI-SearchBot: blocked by 2% of companies, named in 14%.
• ChatGPT-User: blocked by 1% of companies, named in 13%.
• ClaudeBot: blocked by 3% of companies, named in 16%.
• Claude-SearchBot: blocked by 1% of companies, named in 6%.
• anthropic-ai: blocked by 3% of companies, named in 9%.
• PerplexityBot: blocked by 1% of companies, named in 15%.
• Google-Extended: blocked by 3% of companies, named in 14%.
• CCBot: blocked by 4% of companies, named in 12%.
• Bytespider: blocked by 8% of companies, named in 12%.
• Applebot-Extended: blocked by 3% of companies, named in 10%.
• meta-externalagent: blocked by 3% of companies, named in 9%.
• Amazonbot: blocked by 2% of companies, named in 9%.
• 97% of software and AI companies (190 of 196) have a robots.txt file.
• 87% list a sitemap in robots.txt.
• 59% publish an llms.txt file (115 sites).
• 13% also publish llms-full.txt.
• 23% name at least one AI crawler in robots.txt.
• 5% fully block at least one major AI training crawler (GPTBot, ClaudeBot, CCBot or Google-Extended).
• 2% block GPTBot (training) while leaving OAI-SearchBot (ChatGPT search) allowed.
• 90% block none of the 13 AI crawlers we checked.

Structured data and meta tags

Source: AgentLaunch crawl of robots.txt, llms.txt and homepages, October 2026.

• 65% of directories use JSON-LD structured data on the homepage.
• 55% mark up an Organization; 58% a WebSite.
• 6% mark up a SoftwareApplication; 16% an FAQPage.
• 89% have a meta description; 80% a canonical tag; 86% Open Graph tags.
• 69% are served through Cloudflare, whose bot settings can block AI crawlers independently of robots.txt.
• 63% of companies use JSON-LD structured data on the homepage.
• 58% mark up an Organization; 41% a WebSite.
• 18% mark up a SoftwareApplication; 8% an FAQPage.
• 87% have a meta description; 80% a canonical tag; 87% Open Graph tags.
• 48% are served through Cloudflare, whose bot settings can block AI crawlers independently of robots.txt.