What a log line contains
A server access log is one line per request, always the same shape. In Apache's Combined Log Format: the requesting IP, a timestamp, the method and path, the protocol, the status code, bytes sent, the referrer, and the user-agent string.
Read one line and you know that at 09:14 on Tuesday something calling itself Googlebot asked for /collections/kurtas?sort=price&page=7 and got a 200. Read a million and you know how Google is spending its attention on your site — status codes in aggregate, crawl frequency by directory, and the first date each URL was ever fetched.
Getting the logs — the genuinely annoying part
This is where most log analysis projects die, so deal with it first. Where your site is hosted determines whether this is a ten-minute job or a procurement conversation.
| Setup | How you get logs | The catch |
|---|---|---|
| Shared cPanel (Hostinger, BigRock, GoDaddy India, Bluehost) | cPanel → Raw Access Logs → download the gzipped file. | Often only the last 24 hours. Turn on log archiving first, then wait a week. |
| VPS or dedicated (Nginx / Apache) | /var/log/nginx/access.log and its rotated .gz siblings. | Rotation may discard them after 7 or 14 days. Check logrotate before you need them. |
| Vercel | Dashboard runtime logs are short-lived; a Log Drain streams them somewhere you control. | Drains sit on paid plans. Set one up before the migration, not after. |
| Cloudflare in front of anything | Logpush to R2, S3 or a log platform. | Enterprise feature. Lower plans get sampled analytics, not raw lines — and cached responses never reach your origin log. |
| Shopify, Wix, Squarespace | You can't. There is no raw log access. | Use Search Console's Crawl Stats report, and accept that it's aggregated and sampled. |
Verify Googlebot before you believe it
The user-agent string is self-declared text. Anyone can send it, and plenty of scrapers do, precisely because sites whitelist Googlebot.
The check is two DNS lookups. Reverse-resolve the IP — the hostname must end in googlebot.com or google.com. Then forward-resolve that hostname and confirm it returns the same IP. Anything that fails either step isn't Google. For a million-line file, match against the crawler IP ranges Google publishes as JSON instead.
Do it before analysing anything, or you'll spend a morning worrying about crawl budget burned by a scraper in Frankfurt.
The three questions logs answer that no crawler can
Screaming Frog and Sitebulb simulate a crawl: they tell you what a bot *would* find starting from your homepage today. That's a hypothesis. Logs are the record. Aggregate a week of verified Googlebot hits by directory and three things become answerable.
- What is Google spending its crawl on? Group requests by path pattern. If 60% of hits land on
?sort=and?colour=variants of twelve collections, your faceted navigation is eating the budget your product pages needed. That's arobots.txtand canonical job — and now you can justify it. - How often does each template get recrawled? Money pages daily and blog archives monthly is healthy. Money pages monthly means your internal linking is starving them, and new content won't fix that.
- What has Google never requested? Diff your sitemap against the URLs appearing in the logs. Anything in one and absent from the other has never been fetched — usually because nothing links to it. That's an orphan page list you can't build any other way.
How to actually run one
You don't need a platform. A spreadsheet and a filter cover a small site; Screaming Frog's Log File Analyser handles a few million lines on a laptop and imports a crawl alongside them.
The sequence: pull at least seven days, verify Googlebot, drop everything else, group by directory and status code, then join against a fresh crawl and your sitemap. The findings write themselves.
Cadence: once a year for a brochure site, quarterly for a large catalogue, and always in the fortnight after a migration — that's when logs are worth most, because they show which old URLs Google is still asking for and getting a 404 from. It belongs in any technical SEO audit above a few thousand URLs, and it's overkill below that.