How to run a log file analysis without giving up at step one

The half of this that nobody writes about

Every guide to log file analysis opens with a sample log line and a pivot table. That is the easy half. The hard half is that on the day you decide to do this, the file you need either does not exist yet, was deleted at 04:00 this morning, or contains a Cloudflare IP address on every single row.

We run this often enough on client sites to have a rule of thumb: budget one afternoon for the analysis and up to a fortnight for the extraction. The fortnight is not work. It is waiting — for a support ticket to be answered by someone who has never been asked this before, for a logrotate change to accumulate seven days of data, for the one person with root access to come back from leave.

So this page runs in the order the job actually runs. Where the file lives, how long you have before it is gone, what a CDN does to it, which rows to throw away, and only then what to look for. The definition and the anatomy of a log line already sit on the log file analysis glossary page — this is the procedure that comes after it.

Where the file actually is, by stack

Almost every Indian site we look at sits on one of six arrangements. Find yours, run the move in the third column, and you will know within ten minutes whether this project is a download or a procurement conversation.

• Ask for one domain, one date range, in one email. Hosting support answers "please send the raw access logs for example.com from 1 to 14 September, gzipped" far more reliably than a description of what you are trying to achieve.
• Check the file size before you celebrate. A 2 KB file for a fortnight means the format is wrong, the wrong vhost was exported, or the site is being served entirely from cache.
• Get the error log too if it is offered. It costs nothing and it is where 5xx responses explain themselves.

Rotation: you have less time than you think

A log is not an archive. It is a rolling buffer that a scheduled job empties, and on most Indian shared hosting the default is that yesterday has already gone.

The mechanism is the same everywhere. Apache's own documentation recommends piping CustomLog through rotatelogs on a fixed cycle — commonly 86400 seconds, so one file per day — or handing the job to logrotate with a graceful restart. Nginx on Debian and Ubuntu ships a logrotate config that runs daily and keeps a fixed number of generations. Hosts then edit that number downwards, because disk costs money and nobody complains about missing logs until the week they need them.

Two consequences follow. The first is scheduling: a log analysis is not something you decide on Tuesday and finish on Tuesday. The second is worse. The fortnight in which logs are worth most is the fortnight after a migration, a replatform or a hack — and that is precisely the period nobody thought to prepare for.

1. Find out what you hold today. On cPanel that is whether archiving is switched on. On a VPS it is ls -la in the log directory, then the rotate and daily values in the relevant logrotate config.
2. Raise it to at least 30 days, and 90 if the disk allows. An access log line runs a few hundred bytes before compression and roughly a tenth of that after, so three months is usually a small fraction of storage you already pay for. Measure your own file sizes rather than trusting that estimate.
3. Fix the IP field in the same change window if a CDN sits in front of you. Ninety days of logs that all say the request came from 172.68.x.x are ninety days of nothing.
4. Write the retention number into your runbook, and re-check it after any server rebuild. A fresh image silently resets it to the distribution default.

Your CDN writes its own IP on every line

This is the fault that costs the most time, because the file arrives, opens, and looks entirely normal. Every row has an address, a timestamp, a path, a status code and a user-agent. The addresses are simply all wrong.

A reverse proxy terminates the connection. Your origin server sees the proxy, not the visitor, so the first field on every line in Apache's combined format records a CDN edge IP. Every row, including Googlebot's. You can still read paths and status codes, but you cannot verify a single crawler, which means you cannot trust a single attribution.

The proxy passes the real address in a header instead. Cloudflare's CF-Connecting-IP header carries the client IP that connected to Cloudflare, and Cloudflare's own guidance is to read that rather than X-Forwarded-For, which can contain a comma-separated list of addresses. So the fix is a log format change on the origin, not a plugin.

• Apache — add %{CF-Connecting-IP}i to your LogFormat, or enable mod_remoteip with RemoteIPHeader CF-Connecting-IP and log %a instead of %h.
• Nginx — add $http_cf_connecting_ip to your log_format, or configure the real-IP module with real_ip_header CF-Connecting-IP and Cloudflare's published ranges in set_real_ip_from.
• Add the ray ID while you are in there — %{CF-Ray}i or $http_cf_ray. Cloudflare recommends putting it in origin logs so a row can be matched against a request Cloudflare saw.
• Then wait. The change applies going forward only. Everything already written stays wrong, and no amount of processing recovers it.

Throw away the rows that are lying to you

The user-agent string is text the client chooses. Anyone can send Googlebot/2.1, and plenty of scrapers do, precisely because sites and firewalls treat that string generously.

Google's documented check is two lookups in a fixed order, and one on its own is not enough — a reverse DNS record can be set by whoever controls the address block, so a forged hostname is trivial. The forward lookup is what makes the check hold.

• Run the filter before you count anything. Otherwise a scraper in Frankfurt becomes an afternoon's worry about crawl budget, and a genuine finding stays buried underneath it.
• Keep the discarded rows in a separate file. Fake-Googlebot volume is its own signal, and it is the honest answer when someone asks why bandwidth is high.
• Split verified Googlebot by user-agent. Smartphone and desktop are separate strings, and under mobile-first indexing the smartphone crawler is the one whose behaviour matters.
• Expect to remove something. We won't put a percentage on it, because it varies enormously with how attractive your site is to scrapers. What we can say is that we have never run this check and found nothing to discard.

1. host <ip> — the hostname returned must end in googlebot.com, google.com or googleusercontent.com. Anything else is not Google, whatever the user-agent claims.
2. host <that hostname> — it must resolve back to the address you started with. If it does not, the reverse record is forged and the row goes.
3. For a file with millions of rows, do not do this per line. Google publishes its crawler IP ranges as JSON in CIDR form; match the address column against those in one pass and you get the same answer without a DNS query per request.

The three crawl-waste patterns worth acting on

Once the file is clean, you can find dozens of things. Three of them earn a ticket. The rest are interesting, and interesting is not the same as worth a developer's week.

• Noise: image, CSS and JavaScript requests. They are most of the file by row count and almost never the finding. Filter them out first unless you are specifically chasing a rendering problem.
• Noise: a single day's spike. Crawl rate is bursty. One heavy Tuesday is not a pattern. A fortnight is.
• Noise: AI crawlers, unless bandwidth is the question. GPTBot, ClaudeBot and PerplexityBot are visible in the file and genuinely interesting, and they tell you nothing whatsoever about how Google crawls you. Whether to block them is a separate decision on a separate timescale.
• Noise: low crawl frequency on a small site. Google's own crawl budget guidance starts at roughly 10,000 pages with daily changes, or a million pages overall. Below that, crawl budget is not your constraint and the logs will not tell you anything your rankings haven't.

What to change, and how to prove the change worked

A log analysis that ends in a document is a document. The output should be three or four tickets with owners and dates, and a stated expectation of what the next pull will look like.

The proof step is the one people skip, and it is the reason log analysis has a reputation for being an expensive way to feel technical. Pull a second window after the fix, the same length as the first, verified the same way and grouped the same way. Then compare shares, not absolute counts — Google's overall crawl volume on your site moves for reasons that have nothing to do with anything you did, so an absolute drop in requests to parameter URLs proves nothing on its own.

Give it four weeks. A robots.txt change takes effect once Google re-reads the file, which it caches for up to 24 hours, but the redistribution of attention across a large site takes considerably longer than that.

1. Pull at least 14 days, ideally 30, from the stack you identified above.
2. Confirm the first field is the real requester and not your CDN. If it isn't, fix the format and come back in a fortnight.
3. Verify Googlebot by reverse and forward DNS, or against the published CIDR ranges, and separate the rows that fail.
4. Drop static assets, then group by path pattern with query strings collapsed, and by status code.
5. Diff the distinct URL list against your sitemap in both directions.
6. Write three or four tickets. Name the pattern, the share of crawl it accounts for, the fix, and who ships it.
7. Pull the same window length four weeks later, compare shares, and cross-check the direction against the Crawl Stats report in Search Console.

When this is worth doing, and when it is not

We turn this work down more often than we sell it, which is unusual enough to be worth explaining.

Below a few thousand URLs, Google fetches everything it wants to. Crawl budget is not the constraint; content and links are. Running a log analysis on a 300-page site produces a tidy report, an invoice, and no change in outcome — and we would rather say that than take the money.

Above that, three situations justify it every time. A large catalogue with faceted navigation, where the parameter question is genuinely unanswerable any other way. The fortnight after a migration or a replatform, at any size, because logs are the only record of what Google is still asking for. And a site where a meaningful share of published pages have never been indexed, where the sitemap diff answers in an afternoon what six months of guessing has not.

It belongs inside technical SEO work rather than sold as a standalone deliverable, because the findings are worth nothing until somebody ships the fix. If an agency offers you a log file analysis as a product with a price and no accompanying development, ask who is doing the second half.