Which faceted URLs deserve to exist: a demand-based decision table

The tag argument is downstream of the URL argument

Most faceted navigation advice opens with the tag. Canonical or noindex, nofollow the filter links or don't. That argument is real and we've written the canonical versus noindex decision out separately. It is also the second question.

The first is which of these URLs should exist as an indexable page at all. Get that wrong and the tag choice is decoration on a decision already made badly.

A modest catalogue does the arithmetic for you. Take one clothing category with five filter groups — size with 8 values, colour with 12, brand with 20, price band with 6, material with 5. The number of combinations the front end will cheerfully generate is 8 × 12 × 20 × 6 × 5 = 57,600 URLs, before sort order and pagination multiply it again. The category holds maybe 400 products.

Nobody decided to publish 57,600 pages. The front end did it, because filters compose and URLs are free. That is the actual problem here: an information architecture nobody authored, inherited by a team that now has to work out which parts of it were meant.

The demand test, and the threshold we use

The rule is demand. A filter combination earns an indexable URL when someone searches for it in roughly those words, and when the page they land on is worth the click.

Three conditions, all of them, before we index a facet.

1. Somebody types the phrase. Our floor is roughly 50–100 searches a month in India for the head phrasing plus its close variants. The range is wide on purpose: keyword tools disagree by a factor of two or three at low volumes, and Indian volumes are systematically understated because a chunk of the demand arrives in transliterated or mixed-language phrasing that gets bucketed elsewhere. Below that floor you're building a page for a handful of clicks a month and paying for it in crawl, template maintenance and internal links you had to spend somewhere.
2. There is stock behind it. At least ten products, and it has to stay above ten through a normal stock cycle. A facet page that empties out is a thin page that used to rank. Google's own guidance is to return a 404 when a filter combination has no results, which tells you plainly what it thinks of the empty-shell alternative.
3. A customer would say it out loud in one phrase. "Cotton kurta sets" passes. "Cotton kurta sets under ₹1,499 in green, size M, newest first" does not. If you can't say it without a comma, it isn't a search.

The decision table

Four treatments. Each does a different job, and the trigger for each is a property of the URL rather than a preference about tags.

Why the crawl block is the last step, not the first

Almost every site we look at reached for Disallow: /*?*filter= first. One line, free, and it feels like it deletes the problem.

Google does recommend robots.txt for facets — but as prevention, on a facet space you've decided will never need indexing. Applied as cleanup to a facet space Google already holds, it fails in three specific ways, and all three are mechanical rather than debatable.

1. It doesn't remove anything. Disallow controls crawling, not indexing. Google's documentation is blunt about it: robots.txt "is not a mechanism for keeping a web page out of Google", and a disallowed URL can still be indexed if something links to it. In Search Console these surface as Indexed, though blocked by robots.txt — a warning about a state you created, not a fix.
2. It freezes the junk in place. For noindex to work, Googlebot has to fetch the page and read the rule. Block the URL and the crawler never sees it, so URLs already in the index stay there — indefinitely, because nothing will recrawl them to learn otherwise. This is the failure that catches teams out: both changes ship in the same sprint and cancel each other.
3. The pattern is blunter than the decision. A disallow pattern matches strings, not intent. Disallow: /*?*colour= catches the 400 junk combinations and also catches ?colour=black, which was the one facet that cleared your demand threshold. Now the page you wanted is uncrawlable and the ones you didn't are still indexed.

An indexable facet nobody can reach is not a page

This is the half that gets cut when the sprint runs long. Deciding a facet URL deserves to be indexed is a decision about a URL. Making it worth anything is a decision about links.

Most filter interfaces don't produce links at all. They produce click handlers that rewrite the address bar and repaint the product grid. There's no <a href> for a crawler to follow, so the URL you carefully chose to index is an orphan: it exists, it may sit in a sitemap, and nothing on your site points at it.

• Real anchors for the ones that matter. The indexable combinations need actual markup links — /kurtas/cotton, in the HTML — not a widget that mutates the URL on click. The junk combinations can stay JavaScript-only. That isn't a compromise, it's the design.
• A curated module, not the whole filter set. A short block on the category page listing the eight to fifteen facet URLs that cleared the demand test. That is the entire internal linking plan, and it's an afternoon of developer time.
• Its own title, H1 and a real paragraph. If the only difference between /kurtas and /kurtas/cotton is which products the grid returns, you built a duplicate and Google will file it as one. Two or three sentences of genuine copy about the subcategory is the floor, not the aspiration.
• Sitemap entries only for the indexable set. A sitemap full of URLs you have noindexed is a signal you don't want to send. And the anchors matter more than the sitemap entry — see how internal linking distributes value for why.
• Breadcrumbs that go up. The facet page should link back to its category, not sideways into forty sibling filters. Sideways links are how you rebuild the maze you just dismantled.

Sizing the crawl waste so the change is provable

Do this before you touch anything, because afterwards nobody will believe the number.

Start with the honest caveat: crawl budget is a genuine constraint on a small minority of sites. Google's rough classification is sites over a million pages that change weekly, or over ten thousand pages that change daily. If you run a 600-SKU D2C brand, faceted navigation is still worth fixing — but the reason is index quality and where your internal links go, not crawl. Crawl budget covers why that distinction gets sold dishonestly.

1. Crawl Stats, in Search Console. Settings, then Crawling, then Crawl stats. Ninety days of requests broken down by response, file type, purpose — discovery versus refresh — and Googlebot type. It's available only on a root-level property, a domain property or a URL-prefix property at the root, which is the first thing people trip over.
2. The one number worth writing down. The share of Googlebot's HTML requests hitting a URL that contains a filter parameter. Crawl Stats won't slice it that way, so read it from server logs if you can get them — a single week is enough for the ratio. Record it with the date beside it.
3. Index bloat, from the Page indexing report. Count the URLs sitting in Crawled — currently not indexed and Duplicate without user-selected canonical. On a site with an open facet space, those two buckets are usually where the tens of thousands live.
4. Re-measure at four weeks and eight weeks. What you want is the crawl mix shifting toward product and category URLs, and those two Page indexing buckets shrinking. Traffic is not the first thing to move, and pretending it will be sets you up for a false failure at week three.

The order we run this in

Sequence matters more than any individual choice here, because two of these steps undo each other when they land in the wrong order.

1. Crawl the site and export every URL the front end can generate. Group by parameter and count the URLs in each group. This is the map nobody has.
2. Pull demand data for the combinations that look plausible — usually thirty to sixty candidates. Most will fail the threshold. That's the expected result, not a sign you researched it badly.
3. Write the list of facet URLs that will be indexable. On a typical Indian D2C catalogue it lands somewhere between eight and forty, not four hundred.
4. Ship the template work for that list: title, H1, intro copy, breadcrumb, self-referencing canonical.
5. Ship the internal links — the curated module on each category page, with real anchors in the markup.
6. Apply noindex, follow to everything outside the list that is currently indexable, and confirm nothing in robots.txt is blocking those URLs. If something is, remove the block first and wait for a recrawl.
7. Wait, and watch the Page indexing report. Eight to sixteen weeks for the bulk of a large facet space to clear.
8. Only then, and only if crawl is genuinely a constraint at your scale, add the Disallow patterns — written narrowly enough that they cannot match anything on the indexable list.