Glossary
What is crawl budget?
Crawl budget is the number of URLs a search engine is able and willing to crawl on a website in a given period, set by how much load the server can handle and how much the search engine wants to revisit the site’s pages.
What sets it
Google describes crawl budget as two parts. The crawl capacity limit is how fast Googlebot can fetch pages without slowing your server; fast, error-free responses raise it. Crawl demand is how much Google wants to crawl, based on how popular your pages are and how often they change.
Other search engines and AI crawlers have their own limits, but the same logic applies: fast servers and useful, changing pages get crawled more.
When it matters
For most B2B websites, with a few hundred or a few thousand pages, crawl budget is not a real constraint. Google’s own guidance says it mainly concerns very large sites, roughly a million or more pages, or sites with tens of thousands of pages that change daily.
What smaller sites do suffer from is wasted crawling: bots spending their visits on junk URLs while new pages wait.
How B2B sites waste crawls
- Endless filter and sort combinations on resource or template listings
- Tracking parameters creating new URLs for the same page
- Long redirect chains and links to pages that return 404
- Old tag, author and archive pages with thin content
- Slow server responses that make crawlers back off
Example
A company with 300 real pages adds a resource library whose filters create thousands of URL combinations. Crawlers spend their time on those, and a newly published case study takes weeks to be picked up. Blocking the filter URLs in robots.txt and linking the case study from the home page fixes both.
How we handle it
We keep URL structures clean, avoid crawlable filter combinations, keep internal links pointing at final URLs and list only real pages in the sitemap. On large migrations we also help search engines find the new URLs quickly with updated sitemaps and IndexNow.
Last updated 15 November 2026
