The conflict we find in almost every audit
Someone adds noindex to a set of pages — internal search results, filtered listings, an old campaign directory. Later, someone else notices Google crawling thousands of those URLs and adds a Disallow to robots.txt. Both changes look correct alone. Together they break.
The mechanic: robots.txt controls crawling, the meta robots tag controls indexing, and the tag lives inside the page. Block the crawl and Google never fetches the HTML, so it never reads the noindex. The URL stays in the index — usually stripped of its description — and can sit there a long time.
The fix is counter-intuitive and always the same: remove the robots.txt block. Let Google crawl the pages, read the noindex, and drop them. Add the disallow back afterwards if crawl waste still worries you.
Two ways to deliver it
There are exactly two supported methods, and the choice depends entirely on what kind of file you're dealing with.
- Target one crawler with
<meta name="googlebot" content="noindex">. - The tag must be in the raw HTML
<head>. Injected by JavaScript it works only if Google renders the page, in a second pass you don't control — see JavaScript SEO. - Don't set both methods with conflicting values. Google takes the most restrictive directive, which is rarely what the person who set the looser one intended.
| Method | Where it goes | Use it for |
|---|---|---|
<meta name="robots" content="noindex"> | Inside the <head> of the HTML. | Normal web pages. Easiest to set per-page from any CMS. |
X-Robots-Tag: noindex | An HTTP response header, set at the server or CDN. | PDFs, images, spreadsheets, API responses — anything with no <head>. Also useful for applying rules by path across thousands of URLs at once. |
Does `noindex, follow` still do anything?
It was a standard trick: keep a page out of the index while its links keep passing signals through the site. Paginated archives, tag pages, filtered views.
Google's own line is that it decays. Once a page is reliably noindexed, Google crawls it less, and eventually treats it much like noindex, nofollow — the links stop being followed in practice. Nobody outside Google can give you a timeline.
So don't build architecture on it. If a page is the only route into a section of your site, link that section from somewhere indexable instead — otherwise you're constructing orphan pages on a delay.
Which pages actually deserve it
The honest test: would a searcher be annoyed to land here from Google? If yes, noindex it. If the page is merely a duplicate of one you do want to rank, you want a canonical instead.
- Internal search results. Google's guidelines have flagged these for years, and
/search?q=URLs multiply forever. - Thank-you and confirmation pages. Also stops fake conversions from direct landings.
- Cart, checkout and account pages. Nothing to rank, plenty to leak.
- Staging environments — though HTTP authentication is better, because it blocks people too.
- Thin filter combinations — colour plus size plus price, generating thousands of near-identical pages.
How long removal takes, and the launch-day disaster
Removal isn't instant. Google has to recrawl the page to see the directive: days for a frequently crawled URL, weeks for a deep one. If you need something out immediately — a leaked pricing page — use Search Console's Removals tool for temporary suppression while the noindex does the permanent work.
And the failure that costs most: shipping a staging noindex to production. In WordPress it's one checkbox — Settings → Reading → "Discourage search engines from indexing this site" — and it survives the launch more often than anyone admits. Three weeks later someone notices there's no organic traffic at all.
Check it on launch day and again a week later. curl -I the homepage, search the source for noindex, run URL Inspection. Two minutes. It's the cheapest item on any technical SEO checklist and the most expensive one to skip.