# What Is Crawl Budget? When It Matters > Crawl budget is how many pages a search engine will crawl on your site in a given time. What sets it, when it matters, and how B2B sites waste it. Source: https://pentaclay.com/glossary/crawl-budget Glossary # What is crawl budget? Crawl budget is the number of URLs a search engine is able and willing to crawl on a website in a given period, set by how much load the server can handle and how much the search engine wants to revisit the site’s pages. ## What sets it Google describes crawl budget as two parts. The crawl capacity limit is how fast Googlebot can fetch pages without slowing your server; fast, error-free responses raise it. Crawl demand is how much Google wants to crawl, based on how popular your pages are and how often they change. Other search engines and AI crawlers have their own limits, but the same logic applies: fast servers and useful, changing pages get crawled more. ## When it matters For most B2B websites, with a few hundred or a few thousand pages, crawl budget is not a real constraint. Google’s own guidance says it mainly concerns very large sites, roughly a million or more pages, or sites with tens of thousands of pages that change daily. What smaller sites do suffer from is wasted crawling: bots spending their visits on junk URLs while new pages wait. ## How B2B sites waste crawls - Endless filter and sort combinations on resource or template listings - Tracking parameters creating new URLs for the same page - Long redirect chains and links to pages that return 404 - Old tag, author and archive pages with thin content - Slow server responses that make crawlers back off ## Example A company with 300 real pages adds a resource library whose filters create thousands of URL combinations. Crawlers spend their time on those, and a newly published case study takes weeks to be picked up. Blocking the filter URLs in robots.txt and linking the case study from the home page fixes both. ## How we handle it We keep URL structures clean, avoid crawlable filter combinations, keep internal links pointing at final URLs and list only real pages in the sitemap. On large [migrations](https://pentaclay.com/services/website-migration) we also help search engines find the new URLs quickly with updated sitemaps and IndexNow. Read next - [Website migration](https://pentaclay.com/services/website-migration) Last updated 15 November 2026 Related terms ## Terms that go with crawl budget - [robots.txt robots.txt is a plain text file at the root of a website (yoursite.com/robots.txt) that tells crawlers, such as Googlebot or AI bots like GPTBot and ClaudeBot, which parts of the site they are allowed to fetch.](https://pentaclay.com/glossary/robots-txt) - [Sitemap A sitemap is a file, usually sitemap.xml at the root of a website, that lists the URLs a site wants search engines to find, often with the date each page last changed.](https://pentaclay.com/glossary/xml-sitemap) - [Internal linking Internal linking is the practice of linking from one page of a website to other pages on the same website, so visitors and search engines can find related content and understand which pages matter most.](https://pentaclay.com/glossary/internal-linking) - [IndexNow IndexNow is an open protocol that lets a website notify participating search engines, such as Bing and Yandex, the moment a URL is added, updated or deleted, instead of waiting for them to crawl the site.](https://pentaclay.com/glossary/indexnow) [All glossary terms](https://pentaclay.com/glossary) ## Questions about crawl budget ### Can’t find your answer? Book a call and let’s talk. [Book a call](https://cal.com/pentaclay/discovery-call) Or email us at [contact@pentaclay.com](mailto:contact@pentaclay.com) Do I need to worry about crawl budget? Usually not, unless your site has hundreds of thousands of URLs or changes a large share of them every day. For smaller sites, focus on clean URLs, working links and a correct sitemap. How can I see how Google crawls my site? Google Search Console has a Crawl stats report under Settings. It shows requests per day, response times and the share of responses that were errors or redirects. Does blocking pages in robots.txt save crawl budget? Yes. Blocked URLs are not fetched, so crawlers spend their visits elsewhere. Do not block pages you want indexed, and do not use robots.txt to remove pages from search results. Does a faster site get crawled more? Generally, yes. Google says that when a site responds quickly and reliably, its crawl capacity limit goes up. ## Your business grew up. Your website should too If customers need a demo just to understand what you sell, your website isn’t doing its job. - We sign an NDA on request. - You own the site, the code and every account once it’s paid. - Your first call is with Erfan, the founder. **Kazi Mohammed Erfan** Founder Don’t want to fill in the form? [Book a call directly](https://cal.com/pentaclay/discovery-call) ## Not sure? Ask AI about Pentaclay ### Pentaclay, in short Updated 17 Oct 2026 Pentaclay is a B2B website design agency for AI companies that sell to enterprises. We write, design and build marketing websites in Framer, Webflow and Next.js, and stay to help after launch. Pentaclay LLC is registered in New Mexico, USA; our team works from Dhaka, Bangladesh, with calls until 12:30pm US Eastern. Founded in 2023 by Kazi Mohammed Erfan, who brings 12 years of design experience.