Glossary
What is crawl budget?
Crawl budget is the number of URLs a search engine can and wants to crawl on a site in a given time. It depends on how fast and healthy the server is (crawl capacity) and how much the engine wants the content (crawl demand). Google says it mainly matters for very large sites.
The two parts
- Crawl capacity: how much crawling your server can take without slowing down. Fast, error-free responses raise it.
- Crawl demand: how much Google wants to crawl, driven by popularity and how often content changes.
When it matters
Google's documentation targets sites with very many pages (around a million or more, or 10,000+ that change daily). A site with a few hundred or thousand pages usually doesn't need to worry, though slow servers and messy URLs still delay discovery.
What wastes it
| Waste | Fix |
|---|---|
| Faceted and parameter URLs | Block or canonicalise |
| Redirect chains | Link to final URLs |
| Soft 404s | Return real 404s |
| Duplicate pages | Canonicals, consolidate |
| Slow responses | Faster hosting, caching |
Example
An e-commerce site with filter combinations creates two million crawlable URLs for 20,000 products. Blocking filter parameters from crawling lets Google spend its time on product pages, which get recrawled and updated in results much faster.
FAQ
Do small sites need to worry about crawl budget?
Usually not. Focus on internal links and quality.
Does a sitemap increase crawl budget?
No, but it helps Google prioritise what to crawl.
How do I see crawl activity?
Search Console's Crawl stats report.
Sources
- Google Search Central: Crawl budget management for large sites
- Google Search Console Help: Crawl stats report