What counts as bloat, and what is just a big site
A large site is not a bloated one. An ecommerce catalogue with 60,000 genuine product pages is fine. Bloat is the gap between the number of URLs Google has indexed and the number of pages you would be happy for a stranger to land on from a search result.
The quickest read is a ratio. Count the pages you deliberately published — products, categories, articles, service pages, the handful of legal pages. Then look at the indexed count in the Pages report. If the indexed number is four or five times the deliberate number, something is generating URLs on your behalf, and it is almost never a person.
This matters for two reasons, and only one of them is the one people quote. The commonly cited reason is crawl efficiency: Google's own guidance on crawl budget warns that time spent on URLs it shouldn't be crawling is time not spent on the rest of your site. That's real, and on a site under a few thousand URLs it's usually not your bottleneck.
The reason that bites smaller sites is dilution and self-competition. A tag archive that lists three post excerpts is a thin page competing with the posts themselves. Twelve of them competing for the same phrase is a keyword-cannibalisation problem you never briefed. That's the damage, and it lands well before crawl budget does.
Read the Pages report in this order, every time
Search Console's Pages report groups URLs by what Google did with them. Read the groups in a fixed sequence, because the sequence is what separates a template fault from a linking fault, and those have different fixes.
Open the report, switch to All known pages, and work down this list.
- Indexed. Sort by URL and skim the patterns rather than the pages. You are looking for repeated path segments —
/tag/,/author/,/2019/07/,/?attachment_id=— because a repeated segment means a template is producing them. One odd URL is an accident. Four hundred sharing a prefix is a setting. - Crawled – currently not indexed. Google's own definition is that it crawled the page and did not index it, and may or may not do so later. On a bloated site this bucket is usually enormous and usually correct. Google looked at your tag archives and declined. That is not a bug to fix; it's a verdict on the value of those URLs.
- Duplicate without user-selected canonical. Google says this page duplicates another and you gave it no canonical preference, so Google picked one for you. Every URL in this bucket is a place where your site failed to declare which version it wanted, which is the single most fixable category in the report.
- Duplicate, Google chose a different canonical than user. Rarer and more interesting: you declared a preference and Google overruled it. Usually a sign the declared canonical is weaker than the alternative — thinner, slower, or not internally linked.
- Excluded by 'noindex' tag and Blocked by robots.txt. Check these last, and check them for surprises rather than for volume. A page you wanted indexed sitting in either bucket is a separate and more urgent problem — see why Google will not index your page.
- Indexed, though blocked by robots.txt. If this bucket has anything in it, read the robots.txt section below before you touch anything else. It is the specific failure mode this page exists to prevent.
Where it comes from on WordPress and WooCommerce
In India the majority of bloated sites we look at are WordPress, and the bloat is almost always shipped by defaults nobody changed. None of this is a plugin misbehaving. It is the CMS doing exactly what it was configured to do on install day.
The named sources, roughly in order of how often they turn up.
- Shopify's version of the same disease is
/collections/x/products/yduplicating/products/y, and it is handled by the platform's canonical rather than by you. Different symptom, same category of problem. - Faceted navigation is the expensive one on any platform. If your filters build crawlable links rather than firing a request behind the scenes, the URL count is combinatorial and no amount of tidying keeps up.
- Check for a second copy of the whole site while you're here — a staging subdomain, an old
wwwvariant, anhttpversion that never redirected. That's not bloat, it's duplication, and it needs a redirect rather than anoindex.
| Source | What it looks like | Why it multiplies |
|---|---|---|
| Tag archives | /tag/seo-tips/, plus /tag/seo-tips/page/2/ | One tag per idea, three tags per post, and each one paginates |
| Author and date archives | /author/admin/, /2021/08/ | Enabled by default on most themes even with one author and no reason to browse by month |
| Attachment and media pages | /product-shot-2/ or /?attachment_id=4127 | WordPress creates a page per uploaded image unless the theme redirects it |
| Internal search results | /?s=running+shoes | Crawlable the moment anything links to a search URL, and infinite by definition |
| WooCommerce filters and sorting | ?orderby=price, ?filter_size=9, ?min_price=500 | Every combination of every filter is a distinct URL. Ten filters make thousands |
| Cart, checkout and account URLs with parameters | ?add-to-cart=88, /my-account/orders/ | Session and action URLs that should never have been crawlable |
| Feeds and comment pages | /feed/, /comment-page-2/ | One per post, silently, forever |
Choosing the right removal method per source
There are four tools and they are not interchangeable. Choosing by habit rather than by source is why sites get cleaned twice.
The question to ask per URL pattern: does a human ever need to reach this, and does an equivalent page already exist?
- For WordPress specifically: turn off tag, author, date and attachment archives in your SEO plugin rather than editing templates. It's one screen, and it's reversible.
- Google's Removals tool hides URLs for about six months. It is a fire extinguisher for something embarrassing that is live right now, not a fix — the underlying page has to change or it comes back.
- If you are consolidating overlapping articles rather than deleting junk, that's a different job with a different risk profile: how to merge two competing pages covers it.
| Situation | Tool | Why |
|---|---|---|
| A human might reach it, but it should never be a search result — internal search, filtered listings, thin tag archives you want to keep for navigation | noindex in the page's robots meta tag | Drops it from the index while leaving it usable and crawlable, so links on it still pass through |
| It genuinely duplicates another URL — sort orders, tracking parameters, a print view | rel="canonical" to the preferred URL | Consolidates the signals onto one version instead of throwing them away |
| It should not exist at all — attachment pages, an abandoned category, spam URLs from an old hack | 410 Gone, or 404 if 410 is awkward | The clearest possible statement that the URL is finished. Google treats it as a strong signal not to come back |
| It has a real successor — a merged article, a discontinued product with a replacement | 301 redirect, one to one | Moves the accumulated signals rather than dropping them. Never redirect a batch to the home page |
| Vast machine-generated spaces you have already removed from the index — calendar pages, infinite filter combinations | robots.txt disallow, afterwards | Stops the crawling once the indexing question is already settled. See the next section for the order |
Why robots.txt is usually the wrong tool here
This is the mistake that turns a two-week cleanup into a six-month one, and it is made with complete confidence roughly every time.
robots.txt controls crawling, not indexing. Google's documentation is unusually blunt about it: robots.txt is not a mechanism for keeping a web page out of Google, and a disallowed URL can still be indexed if something links to it. That's the whole meaning of the Indexed, though blocked by robots.txt status in your Pages report.
The failure is a sequencing failure. Add noindex to 8,000 tag archives and then block /tag/ in robots.txt, and Google can no longer crawl those URLs — so it never sees the noindex you just added. The pages stay in the index, now with no description, and you have locked the door on the instruction that would have removed them.
So the order is: noindex first, wait for the index count to fall, then block in robots.txt if you also want to stop the crawling. Google's crawl-budget guidance does recommend robots.txt over noindex for genuinely unimportant pages at scale — and it is right, for pages that were never indexed in the first place. For pages already in the index, blocking is how you keep them there. See canonical versus noindex for the equivalent trap on the duplicate side.
How long recovery takes, and what to watch while it happens
Everything here is bounded by recrawl rate. Google has to visit each URL again to notice its new status, and low-value URLs are exactly the ones it visits least often — which is a mildly cruel design when the low-value URLs are the ones you're trying to remove.
Expect a visible trend within two to four weeks on a site with reasonable crawl frequency, and two to three months for the long tail on a large or slow-crawled one. A site with 40,000 junk URLs will not be clean by Friday, and anybody who tells you otherwise is about to run the Removals tool and call it done.
Help it along in the cheap ways: make sure your XML sitemap lists only the canonical URLs you actually want indexed, remove the junk patterns from it, and resubmit. A sitemap is a weak signal, but it's a free one and it points the crawler at the right set.
One expectation to set before you start. Cleaning up bloat rarely produces a traffic jump on its own, because the URLs you removed were mostly earning nothing. What it produces is a site where the pages that matter are easier to find, easier to crawl and no longer competing with a tag archive. Treat a flat traffic line and a falling index count as success.
- Indexed count in the Pages report, weekly. It should fall steadily. A flat line after three weeks means the fix isn't being seen — check for a robots.txt block over the URLs you noindexed.
- The Duplicate without user-selected canonical bucket. It should shrink as your canonicals get picked up. If it doesn't, the canonicals are probably pointing somewhere Google disagrees with.
- Crawl stats, under Settings. Requests to the junk patterns should decline. This is the clearest proof the cleanup worked at the crawl level rather than only at the report level.
- Impressions for your real pages, in the Performance report. Filter to the pages you care about. This is the number that says whether the cleanup bought you anything.
- Nothing at all, for the first ten days. Checking daily is how people panic and revert half the changes before Google has recrawled anything.