The four Google systems that decide your fate
Generic SEO advice talks about 'search engines'. Google is the one that pays your bills in India, and it's specific enough to be worth naming its parts. Each system responds to different inputs, and each one shows you a different report when it goes wrong.
| System | Its job | What you can influence | Where you see it |
|---|---|---|---|
| Googlebot | Fetches URLs on a schedule set by demand and by what your server can handle | Server speed, robots.txt, internal links, sitemap hygiene, total URL count | Settings → Crawl stats |
| Indexing | Decides whether to store a page and which URL represents a set of duplicates | Content quality, duplication, canonical tags, noindex, parameter handling | Pages report; URL Inspection → Google-selected canonical |
| Core ranking systems | Score and order candidate pages for each individual query | Relevance, depth, links, page experience, freshness where the query wants it | Performance report, by query |
| The results layer | Assembles the visible page — packs, features, ads and AI Overviews | Almost nothing directly. Indirectly, by being a strong candidate and being quotable | Falling CTR at a flat average position |
Googlebot: crawl capacity and crawl demand
Google's documentation splits crawling into two constraints, and the distinction is genuinely useful because you fix them differently.
Crawl capacity is how much fetching your server tolerates before responses slow down or start erroring. Googlebot watches response times and error rates and backs off when your site struggles. Slow hosting and intermittent 5xx responses cost you crawling directly.
Crawl demand is how much Google wants your URLs — driven by their popularity and how stale Google's stored copy has become. A page nobody links to on a site nobody visits generates very little demand, which is why new pages on new sites sit undiscovered.
Google states plainly that sites with fewer than a few thousand URLs generally don't need to worry about crawl budget. If you run a 90-page site and pages aren't getting fetched, the cause is discovery or server errors, not budget. Budget becomes real on ecommerce with faceted navigation, on calendars, and anywhere session parameters multiply URLs into the millions.
- Watch the 5xx rate and average response time in Crawl stats. Both directly reduce how much Google fetches.
- Block parameter junk in
robots.txtrather than noindexing it — a URL has to be crawled before Google can see a noindex, so noindex costs crawl budget while robots.txt saves it. - Keep sitemaps to canonical, indexable URLs only.
- Count your indexable URLs before deciding which camp you're in. Most sites guess, and most guess wrong in the same direction.
Index selection: Google picks the canonical, you only suggest
This one surprises people who've been doing SEO for years. rel=canonical is a hint. Google evaluates it alongside internal linking patterns, redirects, sitemap inclusion, HTTPS preference and content similarity, then makes its own call.
URL Inspection shows both values: the user-declared canonical and the Google-selected canonical. When they differ, Google disagreed with you and your page is alive but invisible — it exists, it's just folded into a sibling that shows instead.
That's the first thing to check for any page that 'ranks nowhere' despite being well written and well linked. It's a two-minute check that explains a startling number of mysteries, and the fix is either consolidating the pages properly or making them genuinely different from each other. Canonical tags get used as a duct tape for content duplication, and duct tape is exactly what Google is deciding to ignore.
Core ranking systems, and what a core update actually does
Google publishes a list of its ranking systems, which is more transparency than it gets credit for. The named ones worth knowing: link analysis descended from PageRank, RankBrain and neural matching (working out what a query means when nobody uses those exact words), passage ranking, the reviews system, freshness systems, deduplication, site diversity, and spam systems including SpamBrain and link spam detection.
One structural change matters more than the rest. The helpful content system used to be a separate site-wide classifier you could see arrive and leave. With the March 2024 core update, Google folded those signals into core ranking. A site-wide quality judgement now lives inside the core systems rather than sitting on top as a bolt-on you could wait out.
A core update is a broad re-evaluation, not a punishment. Google's post-update guidance is consistent and famously unhelpful — make good content — but the observable pattern is that sites lose ground because a better answer exists, not because one technical fault was detected. Which means the recovery play is content and relevance, not a frantic hunt for a checkbox you missed.
How AI Overviews choose who to cite
AI Overviews are generated by Gemini models grounded in Google's existing search index. They aren't a separate crawler reading the open web for each question. The summary is written from pages the index already holds, and the citations come from that retrieval.
A question also gets decomposed into related sub-questions, each retrieving its own candidates. That's the mechanism behind the most common confusion here — a page getting cited for a question it never targeted, because it was the strongest candidate for one sub-question buried inside a bigger one.
So the practical work is unglamorous and familiar. Be indexed. Be a strong candidate for the specific sub-questions, not just the head term. Write answers as short, self-contained paragraphs that survive being lifted away from the three paragraphs around them. Keep numbers, prices and specifics in text rather than in an image or a PDF.
And the honest caveat: Google has not published selection criteria, citation sets shift between sessions and users, and nobody has a verified method to guarantee a citation. If someone is selling you one, they're selling a guess. See GEO versus SEO for what's actually different about optimising for this layer.
What Google says publicly versus what the evidence shows
Google's public communication is a mix of genuine documentation and careful positioning, and knowing which is which stops you buying services built on the gap between them.
| What Google says | What the evidence suggests | What to do about it |
|---|---|---|
| There's no single site-wide authority score | Reference documentation exposed in the May 2024 Search content warehouse leak listed site-level attributes, and site-level patterns are visible in practice | Build a site that's coherently about one thing. Don't buy a third-party authority number as a KPI. |
| Clicks aren't used as a ranking signal | Testimony and exhibits in the US antitrust proceedings described click-based systems used within ranking | Don't chase clicks with misleading titles. Do make your snippet an honest, specific promise. |
| Crawl budget isn't a concern for most sites | Broadly true under a few thousand URLs. False the moment faceted navigation multiplies your URL count | Count your indexable URLs, then decide which sentence applies to you. |
| Core updates aren't penalties | Accurate — but a broad re-evaluation is indistinguishable from a penalty when you're reading the revenue chart | Diagnose by query and page, not by update date. |
| AI Overviews send higher-quality clicks | Independent panel data shows fewer clicks overall. Nobody has published data on their quality, so both claims can be true at once | Measure clicks and enquiries on your own site rather than accepting anyone's framing, ours included. |