The one line that has killed more traffic than any algorithm update
A developer sets up staging and, sensibly, adds a robots.txt reading User-agent: * and Disallow: / so Google never indexes the half-built site. Then staging gets promoted to production. The file comes with it.
Nothing breaks. The site loads, the forms work, analytics keeps reporting. Over the following days Googlebot stops fetching pages; over the following weeks the site drains out of the results. By the time anyone connects the traffic graph to a deploy three weeks earlier, recovery is a month of re-crawling.
It's the most common catastrophic technical SEO failure there is, and entirely preventable. Two habits fix it: check yourdomain.com/robots.txt after every infrastructure deploy, and turn on Search Console email alerts so a crawl collapse reaches you in days rather than quarters.
Syntax: four directives, and where the file lives
The file sits at the root of the host — https://example.com/robots.txt, not /blog/robots.txt. It applies per protocol, hostname and port, so a subdomain needs its own. It's plain text, publicly readable, and advisory: well-behaved crawlers obey it, scrapers ignore it. Never use it to hide anything sensitive — you're publishing a map of what you'd rather nobody looked at.
Four directives do almost all the work.
- Specificity beats order. Where an
Allowand aDisallowboth match, Google follows the more specific rule, not the first. - `Disallow:` with nothing after it allows everything.
Disallow: /blocks everything. One character apart, opposite outcomes. - Don't block `/wp-content/`, `/_next/` or your CSS and JS. Google renders your pages. Block the assets and it renders one with no layout, no content and no reason to rank.
- On Next.js, generate it —
app/robots.tsreturning aMetadataRoute.Robotsobject keeps the file in version control.
| Line | What it does |
|---|---|
User-agent: * | Everything below applies to all crawlers until the next User-agent line. |
Allow: / | Permits the whole site. Redundant, but it makes intent obvious to whoever reads this next. |
Disallow: /api/ | Blocks internal endpoints returning JSON nobody should index. |
Disallow: /cart | Blocks cart, checkout and account paths. No search value, and they spawn session URLs. |
Disallow: /*?s= | Blocks internal search results. * matches anything — this is what stops a search box becoming an infinite URL space. |
Sitemap: https://example.com/sitemap.xml | Absolute URL, one line per sitemap. How crawlers that never see Search Console find it. |
Crawling is not indexing — the distinction that trips everyone
Robots.txt controls whether a crawler may request a URL. It says nothing about whether that URL may appear in results. Different systems, and confusing them causes two opposite failures.
Blocked but still listed. If other sites link to a blocked URL, Google can index the URL without ever fetching it. You get a result with a bare URL and no description — Google states outright it has no information about the page. Blocking made it worse.
A noindex that never gets read. Someone adds noindex and blocks the page in robots.txt for good measure. Googlebot can't fetch it, so it never sees the tag, so the page stays indexed indefinitely. The two directives cancel each other out.
The rule: to keep a page out of the index, allow the crawl and use noindex. To save crawl capacity on patterns you never want fetched, block in robots.txt and accept a few may still be listed. Use both on one URL and you get neither.
AI crawlers, and testing what you wrote
Robots.txt is now where you decide who reads your content for something other than search. The agents worth knowing: GPTBot (OpenAI), ClaudeBot (Anthropic), PerplexityBot, CCBot (Common Crawl, which feeds many training sets) and Bytespider. Each takes its own User-agent block.
Google-Extended gets misread most often. It controls whether your content is used for Google's generative AI products. It does not affect Googlebot, indexing or rankings. Blocking it costs nothing in search and costs you presence in answers — a business call rather than a technical one, argued both ways in should you block AI crawlers.
Then test. Search Console's robots.txt report shows the file Google last fetched and any parsing errors — worth checking, because a file returning a 500 is treated very differently from one returning 404. For one URL, use URL Inspection's live test.
What no report tells you is whether the rule you wrote was the rule you meant. If a crawl collapse brought you here, that's technical SEO work — a days-long job, not a months-long one.