LAUNCH
Pricing
Content, SEO & Search

Crawl budget

The number of pages on a website a search engine's automated crawler will visit and index within a given period, which can be a practical constraint for very large websites with many thousands of pages.

The number of pages on a website a search engine's automated crawler will visit and index within a given period, which can be a practical constraint for very large websites with many thousands of pages.

Why it matters

For most small and mid-sized sites, crawl budget isn't a limiting factor — but for large sites, wasting crawl budget on low-value pages (duplicate content, outdated pages that should be removed) can mean important, newer pages get crawled and indexed more slowly than they should.

In a sentence

"We're not large enough yet for crawl budget to be a real constraint, but it's worth monitoring as the site grows well past a few thousand pages."
  • Often confused with: Indexing (crawl budget is about how much of a site a search engine's crawler visits in the first place; indexing is the subsequent step of a search engine actually storing and making a crawled page eligible to appear in search results).