You probably don't have a crawl budget problem
This term gets used to explain traffic drops far more often than it actually explains them. Before spending a week on it, check whether it applies to you at all.
Google's own large-site guidance draws the line clearly: it's written for sites over a million unique pages whose content changes weekly, or sites over about 10,000 unique pages whose content changes daily. On everyone else Google is close to dismissive — if your pages are typically crawled the same day you publish them, this isn't your issue.
A 120-page services site whose new pages sit unindexed does not have a crawl budget problem. It has an indexing problem, an internal linking problem, or a content-quality problem. Chase those first.
It matters because crawl budget work is invisible and slow — an excellent place to hide for three months. If someone hands you a crawl budget plan for a small site, ask what the page count is.
Rate limit and demand — the two halves
Crawl budget isn't a number Google assigns. It's what falls out of two independent constraints.
Crawl rate limit is the ceiling Google imposes to avoid hammering your server. If response times climb or you return 5xx errors, Googlebot backs off automatically. It's server engineering, and it's the half you can move quickly — a slow origin is the commonest reason a large site gets under-crawled.
Crawl demand is how much Google wants your URLs. It rises with links and genuine change frequency, and falls for URLs Google has learned are stale or duplicated. Earned over months, not configured.
The consequence: you can lift your crawl rate by fixing a slow server in a fortnight. You cannot conjure demand for 300,000 near-identical filter URLs, because Google has already decided those aren't worth its time — and it's right.
The two situations where it's genuinely real
Both are common in Indian ecommerce, and both are structural rather than incremental.
A catalogue over roughly 10,000 URLs
A marketplace or large D2C catalogue with tens of thousands of SKUs, plus variants and paginated category pages, genuinely can outgrow the attention Google gives it. The symptoms are specific: new products taking weeks to appear, deep pagination never fetched, a Crawl Stats graph flat while your URL count climbs.
The fix is triage, not more sitemaps. Decide which URLs deserve to exist, consolidate the rest, and keep the ones that matter two or three clicks from the homepage. That's most of what ecommerce SEO is on a big catalogue.
Faceted navigation generating infinite URLs
This one surprises people, because it happens quietly. Take a jewellery collection page with five filters: metal (6 options), stone (9), price band (5), occasion (7), sort order (4). Every combination is a distinct URL — 6 × 9 × 5 × 7 × 4 = 7,560 URLs from one collection page. Across 40 collections, 302,400 URLs, for a catalogue holding maybe 2,000 real products.
Googlebot finds these, because your own filter links point at them. It spends real crawl capacity on combinations no human has ever requested while your new arrivals queue. Add an unbounded date parameter and the space becomes literally infinite.
The fix is to stop generating the links: block the parameter patterns in robots.txt, keep a small set of genuinely useful filter combinations as real pages, and render the rest without producing a linked URL.
What actually frees crawl budget
Open Search Console, Settings, Crawl stats. Take three numbers: total crawl requests over time, average response time, and the discovery-versus-refresh split. A high response time is the fastest fix available. A refresh-heavy split means Google is re-reading old pages instead of finding new ones.
Then, in rough order of return:
- Cut server response time. Googlebot's rate limit responds directly, and unlike everything else here it works within days.
- Block infinite spaces in robots.txt — filter combinations, sort orders, internal search, calendar pages.
- Kill redirect chains. Every hop is a separate fetch. See 301 redirect.
- Keep `lastmod` honest so Google can skip what hasn't changed. Fake dates train it to ignore your sitemap.
- Consolidate thin duplicates. Fewer, better URLs is the only structural fix that helps everything else too.