Crawl budget is the number of URLs Googlebot is willing and able to crawl on your site in a given period. The phrase is repeated in almost every SEO checklist, and most small sites do not need to think about it at all. Knowing which group you are in saves a lot of wasted effort.
The two things that set it
Google describes crawl budget as the combination of two factors:
- Crawl capacity limit. How much Googlebot can fetch without overloading your server. A fast, healthy server raises it. Lots of errors or slow responses lower it.
- Crawl demand. How much Google wants to crawl. Popular pages, fresh content and changed pages create demand. Stale, low-value pages create little.
The budget is roughly what both allow.
Who actually needs to worry
Google's own guidance says crawl budget matters mainly for:
- Very large sites, roughly a million or more unique pages with content that changes about weekly
- Medium to large sites, about ten thousand or more pages, with content that changes daily
- Sites with a large number of URLs in Search Console's "Discovered, currently not indexed" status
If you run a site with a few hundred or a few thousand pages, and new pages get crawled soon after you publish them, crawl budget is almost certainly not your problem. A page that is not indexed is far more often a quality, duplicate or noindex problem. Check those first, starting with how to check if a page is noindex.
What wastes crawl budget
On large sites, the same few patterns do most of the damage:
- Faceted navigation and filters that create thousands of URL combinations (colour, size, price, sort order)
- Duplicate URLs from tracking parameters, session IDs, or both
httpandhttpsversions - Soft 404s, pages that say "not found" but return 200
- Long redirect chains, where each hop costs a request. See 301 vs 302 redirects and redirect chains.
- Infinite spaces, such as calendars with a "next month" link that never ends
- Slow or error-prone servers
How to check how Google crawls you
Search Console Crawl Stats. Under Settings, open Crawl stats. It shows total crawl requests per day, average response time, and a breakdown by response code, file type and purpose. Look for spikes in 404s, 5xx errors or redirects.
Server logs. The most accurate source. Filter for Googlebot to see exactly which URLs it requests, and how often. If it spends most of its time on filter URLs and almost none on your product pages, you have found a real problem.
Compare crawled to wanted. If Google crawls thousands of URLs you never wanted indexed, that is wasted budget.
What to do about it
- Make the server fast and stable. It raises the crawl capacity limit.
- Remove or consolidate duplicates with redirects and canonical tags. See how to check a canonical tag.
- Block crawl-wasting sections in robots.txt, such as internal search and filter combinations. See how to check robots.txt.
- Return the right status codes. A removed page should return 404 or 410, not 200.
- Keep your sitemap clean, listing only URLs worth crawling.
- Link to important pages from strong pages, so crawlers reach them quickly.
Notice that none of these tricks needs the term "crawl budget". They are all just good technical hygiene, which is the real answer for most sites.
Takeaway
Do not chase crawl budget on a small site. Do keep the site tidy: fast server, clean URLs, correct status codes, an accurate sitemap. Those fixes help every crawler and every visitor.