Web & SEO

Crawl Budget

A search engine’s practical capacity and willingness to crawl URLs from a site over time. It matters most to very large or frequently changing sites; smaller sites should usually focus first on crawlable links, correct responses and useful pages.

How it works

Crawl budget describes the amount of crawling a search engine allocates to a site within a period. It is especially relevant to large or frequently changing inventories. Server capacity, crawl demand and wasted URL spaces all matter. It is not a quota that guarantees indexing for every fetched page.

A realistic example

A filter that creates endless combinations of otherwise identical term lists can expose many low-value URLs. Stable canonical term pages and useful normal links make the actual inventory clearer. Fix slow or failed responses before assuming the crawler simply needs a bigger allowance.

What to check

Compare the intended inventory with crawlable parameter URLs, redirects and errors. Inspect server response behaviour and crawl reports where available. Identify patterns rather than judging one missed page in isolation; a small glossary’s indexing problem may be content or canonical selection instead.

Limits and next steps

Do not block essential resources or hide useful pages to chase a crawl number. Remove unnecessary duplicate paths and keep genuine modification signals accurate. A cleaner inventory can aid crawling without promising that every URL will be indexed.

Technical sources

← All glossary terms