The URL count nobody budgeted for
Take 40 categories, 60 cities, eight filters and four sort orders. That is not 40 pages. Multiplied out, with pagination on top, it's a number with six digits in it — and every one of those URLs is a request Googlebot may or may not choose to make.
Marketplaces hit the wall at a predictable moment. Somewhere past the first hundred thousand URLs, indexation stops keeping pace with publication. New supplier pages sit in 'Discovered — currently not indexed' for weeks. Search Console starts reporting more excluded pages than valid ones. The engineering team can't find a bug because there isn't one. The site got bigger than its crawl allocation, and the average quality of a page fell below the line at which Google bothers.
Index thresholds: the rule that decides which pages exist for Google
The useful question is not 'should we index listing pages'. It's 'what has to be true about a listing page before it earns a place in the index'. Write that rule down, put it in the template, and let the code enforce it — because a rule that needs a human to flip a flag will not be applied at 200,000 URLs.
- Never block and noindex the same URL. A page disallowed in robots.txt can't be fetched, so its noindex tag is never seen and the URL can linger in results. This is the most common indexation bug on large sites, and it's usually about a year old by the time anyone finds it.
- Make the thresholds dynamic in both directions. Pages fill and pages empty. A city that drops below the floor because three suppliers churned should quietly step out of the index.
- Don't index a page you wouldn't send a paid click to. It's a crude test and it's right more often than a committee.
| Template | Default | The rule |
|---|---|---|
Category (/plumbers) | Index | Always. This is the head term and the hub everything below it links from. |
Category × city (/plumbers/pune) | Index above a threshold | Index when the page carries enough live listings to be useful. Pick a floor — five to ten is common — and enforce it in code. Below it, keep the page live for users and noindex it until it fills. |
Category × city × filter (/plumbers/pune/24x7) | Index a chosen few | Only the filter values with real search demand: emergency, 24x7, verified, price band. Everything else stays usable but noindexed, or never gets a crawlable URL at all. |
| Sort orders and pagination | Noindex or canonical | Sorting changes order, not content. Noindex the sort variants, self-canonicalise page two onwards, and don't rely on rel=next and rel=prev — Google stopped using them in 2019. |
| Individual supplier or listing page | Index, conditionally | Index when the supplier has real content: description, photos, reviews, hours, coverage. Auto-generated stubs are the single largest source of thin pages on a marketplace. |
| Zero-result pages | Never index | An empty results page is a soft 404 to Google and a dead end to the user. Show the nearest useful alternative and keep the empty state out of the index. |
Segmented sitemaps and log files, or you are guessing
You cannot manage indexation on a marketplace by reading the site-level coverage report. It averages across templates and tells you nothing about which one is failing. Five changes turn that report into something you can act on.
- Split XML sitemaps by template. One for categories, one for city pages, one for suppliers, one per major vertical if you're large. Search Console reports indexation per sitemap, which turns a vague problem into a specific one.
- Cap each sitemap well under the 50,000-URL limit so the percentages stay readable, and set
lastmodhonestly. A sitemap claiming everything changed yesterday gets treated as noise. - Pull server logs monthly and count Googlebot hits by template. Log file analysis is the only way to see where crawl actually goes, and on most large marketplaces the answer is embarrassing: parameter URLs, expired listings and page 14 of pagination absorb most of it.
- Track time-to-index for new supply. Measure the days between a listing going live and its first Googlebot hit, then its first impression. When that number starts climbing you have a crawl problem months before traffic shows it.
- Fix response time before adding pages. Crawl rate is partly a function of how fast you serve. A faster origin genuinely earns more crawl, and it's usually cheaper than another content sprint.
Supply pages before demand: the SEO version of chicken and egg
Every marketplace launches a city before it has suppliers there, and every marketplace asks the same question: publish the empty page and hope, or wait until it fills and lose six months of ranking age?
Neither. Publish it, keep it out of the index, and make it useful with something other than listings — which is exactly the content most marketplaces never write.
- Build the non-listing content first. What the service costs in that city, what a typical job involves, licensing or registration requirements, seasonal demand, how long people usually wait. This ranks on its own and it's what lets the page clear a quality bar the moment listings arrive.
- Seed from the demand side. A city page with zero suppliers and a working 'post a requirement' form still converts, still collects intent data, and tells your supply team exactly where to recruit next.
- Never fake supply. Scraped listings and placeholder profiles are the shortest path to a manual action, and in India the shortest path to a consumer complaint thread outranking your own brand page.
- Promote to indexable the week the floor is crossed, not at the next quarterly review. That gap is where months of ranking age quietly disappear.
Duplicate listings and UGC: the ranking asset that also breaks the site
Reviews, questions and supplier descriptions are the only content on a marketplace a competitor cannot copy. They are also where most of your duplication comes from, because suppliers list themselves three times, copy each other's descriptions, and paste the same paragraph across every city they claim to serve.
- Deduplicate at creation. Phone number, GST number and address matching at listing time prevents a problem that is close to unfixable at 200,000 URLs.
- Enforce a minimum on supplier-written copy, or generate structured content from fields you actually hold — services, coverage, hours, price band, response time — rather than shipping a blank description because a supplier couldn't be bothered.
- Moderate reviews and keep the bad ones. A category where every supplier holds five stars reads as fake to users and to Google, and review markup that only ever shows perfect ratings invites a structured-data manual action.
- Never mark up ratings you didn't collect. Pulling a supplier's rating from their own website and republishing it as yours is a policy problem, not a clever shortcut.
- Retire expired listings deliberately. 410 for genuinely gone, 301 to the parent category where a real equivalent exists, and keep the page alive with a 'no longer available — here are 12 nearby' block where the URL has links. The one thing you must not do is leave 40,000 dead pages returning 200.
When a template turns out to be thin: fill, merge or prune
Sooner or later you will find a template producing 30,000 pages and 400 clicks a month. There are three honest responses, and 'publish more of them' is not one of them.
- Fill — if the pages could genuinely carry different content and you will actually build it. Data you already hold counts: median price in that city, typical response time, how many suppliers, what changed this month.
- Merge — collapse
plumbers/pune/24x7andemergency-plumbers/puneinto whichever URL is stronger, 301 the loser, and stop pretending they were different queries. - Prune — noindex, or remove and return 410. Cutting 30,000 thin pages regularly improves how the survivors perform, because thin content at scale is a site-level signal, not a per-page one.