Server logs are the files a web server writes as it works, with one line for every request it receives. Each line usually records who asked (an IP address), when, for what (the URL), how the server answered (an HTTP status code), how much data went back, and which software made the request (the user agent). They exist because the server is present for every request. Ad blockers stop analytics tags and crawlers never run them, but nothing reaches a site without passing through a server. For marketers, that makes logs the closest thing the web has to a ground-truth record of traffic, including the large share of it that is not human.

What a log line contains

Two kinds of log matter most. The access log records requests. The error log records problems the server ran into, such as a misconfiguration or a script that crashed. Marketing and search work almost always concerns the access log.

The dominant layout is the Combined Log Format. In the Apache HTTP Server documentation, it is defined by the directive LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-agent}i\"" combined. The documentation's own example line reads:

127.0.0.1 - frank [10/Oct/2000:13:55:36 -0700] "GET /apache_pb.gif HTTP/1.0" 200 2326 "http://www.example.com/start.html" "Mozilla/4.08 [en] (Win98; I ;Nav)"

Left to right: client IP, an identity field that is almost always a dash, authenticated user, timestamp, request line (method, path, protocol), final status code, response size in bytes, referring page and user agent. The first seven fields, without referrer and user agent, form the older Common Log Format.

Nginx, the other widely used web server, ships the same structure. According to its documentation, the configuration "always includes the predefined 'combined' format", built from variables such as $remote_addr, $time_local, $request, $status, $body_bytes_sent, $http_referer and $http_user_agent, and it writes to logs/access.log by default.

Microsoft's IIS server and many cloud services instead use the W3C Extended Log File Format, a 1996 working draft by Phillip Hallam-Baker and Brian Behlendorf. A #Fields: directive declares the columns, named with prefixes such as cs (client to server) and sc (server to client).

Volume adds up quickly. The Apache documentation notes that an access log "typically grows 1 MB or more per 10,000 requests", which is why servers rotate logs daily or by size and compress old files.

Where logs sit in the request flow

On a simple site, the log lives on the origin server. Most commercial sites now sit behind a content delivery network (CDN) such as Cloudflare, Akamai, Fastly or Amazon CloudFront, which answers many requests from its own cache. Those requests never reach the origin and never appear in its log. The complete record is therefore the CDN's log, which has to be exported. Cloudflare's Logpush, for example, delivers logs in batches to storage buckets and security tools. According to its documentation, last updated October 30, 2026, Logpush cannot backfill historical data and offers no storage or search of its own.

On the sell side, publishers and hosting platforms own the logs. Advertisers see them only for their own sites and landing pages.

Origin and evolution

The Common Log Format came from the NCSA HTTPd server in the early 1990s and was inherited by Apache, which took over much of that codebase in 1995. In the web's first decade, logs were analytics. Programs such as Analog and Webalizer, and later Urchin, read log files and turned them into reports of hits, pages and visitors.

That role shifted after the early 2000s. Caching by proxies and browsers hid repeat requests from servers, and IP addresses proved a poor stand-in for people. JavaScript tags, which run in the browser and set cookies, counted people more convincingly. Google bought Urchin in 2005 and relaunched it as the tag-based Google Analytics, and log analysis retreated to engineers and technical search specialists.

Two developments revived marketing interest. The first was podcasting, where files are downloaded rather than played in a browser and there is no tag to fire. IAB Tech Lab's podcast measurement guidelines state plainly that measurement "is based on server logs". Version 1.0 appeared in September 2016, version 2.0 in December 2017, and version 2.3 in September 2026. The guidelines require filtering: removing two-byte range requests that players use to probe a file, removing self-identifying bots, and counting one download per IP address and user agent within a 24-hour window, using one of three window methods (calendar day, first touch or last touch).

The second was AI crawling. Tags cannot see bots, so logs became the instrument for measuring them.

Why server logs matter for marketers

Search specialists use logs to see how often Googlebot and Bingbot fetch which pages, which URLs waste crawl budget on redirects and errors, and whether new content is being found. Google's own documentation has framed it bluntly: "When you see Googlebot in your server logs, you are just looking at Google Search," a line covered when Google rewrote its crawler documentation in March 2026.

Since 2024 logs have mostly been used to measure AI crawlers. A study by Chris Long of Nectiv with Botify, drawing on more than 7 billion log entries from November 2024 to March 2026, showed that OAI-SearchBot activity rose 3.5 times after GPT-5 while ChatGPT-User events fell 28%. Kinsta's analysis of more than 10 billion requests found a crawler identifying as ClaudeBot sending 3.75 million requests to a single WooCommerce cart page in 24 hours. Ahrefs, using logs from 137,000 domains, found that 97% of llms.txt files received zero requests in May 2026.

The gap between logs and analytics can be large. One independent site, PatronView, answered 2.5 million requests in a week while its self-hosted analytics counted 5,977 human pageviews, roughly 214 non-human page loads per human one.

Logs also recover human traffic that analytics misfiles. Natzir Turrado's logs showed that the Gemini iOS app opens links without a referrer, so GA4 records those visits as Direct while the server sees a GeminiiOS user agent. For paid search, logs can expose click fraud: Faktica's investigation found single GCLIDs appearing in server logs from multiple IP addresses, evidence that an excluded IP could still generate charged clicks.

Vendors have productised the idea. Microsoft Clarity's Bot Activity dashboard, launched January 21, 2026, relies on server-side log collection through CDN integrations, and Cloudflare's July 2026 dashboard put crawl-to-referral ratios between 118 and nearly 50,000 crawls per referral.

Limitations and disputes

The user agent is self-reported. Changing it is, in the words of Google's Gary Illyes, "literally one line of code", and he estimated it would take "probably 30 seconds" to set up a script that crawls as Googlebot. GreyNoise counted 263,849 forged sessions using the name Google-Extended in August 2026. A log line saying GPTBot proves only that something claimed to be GPTBot.

Verification therefore relies on the IP address. Google's procedure, last updated March 20, 2026, is a reverse DNS lookup on the logged IP, a check that the hostname ends in googlebot.com, google.com or googleusercontent.com, then a forward lookup confirming the hostname resolves back to the same IP. Google also publishes JSON files of crawler IP ranges. Not every operator does: OpenAI's OAI-AdsBot launched in April 2026 without a published IP list, which makes it harder to check.

IP addresses are themselves a problem. Under EU law they can be personal data: the Court of Justice ruled in Breyer (C-582/14) in October 2016 that even dynamic IP addresses held by a website operator can qualify. Logs therefore carry retention, access and security obligations under the General Data Protection Regulation (GDPR), and many operators truncate or hash IPs, which weakens verification. IAB Tech Lab's podcast rules truncate IPv6 addresses to 64 bits for the same reasons.

Logs also miss things. They record delivery, not attention: a 200 response does not mean anyone saw the page, and a podcast download does not mean anyone listened. Cached and CDN-served requests vanish from origin logs. Shared IP addresses, from mobile carriers or corporate networks, merge many people into one, while one device on several networks splits into many. Log studies from hosts, CDNs and SEO vendors reflect each vendor's customer base, so their figures are not interchangeable.

How server logs differ from adjacent terms

  • Web analytics data comes from tags running in the browser. It describes human sessions well and is blind to bots and blocked scripts. Logs record every request reaching the server but describe people poorly.
  • Server-side tagging sends analytics and conversion events from a server a company controls to vendors such as Google or Meta. It is a data pipeline, not a record of raw requests.
  • Crawl Stats in Search Console is Google's own summary of how Googlebot fetched a site. It covers only Google, and only what Google chooses to report.
  • Referrer data is one field inside a log line, the Referer header, and is often stripped by apps and browser privacy policies before it reaches the server.

Recent developments

Google moved its crawler IP range files to a new path in March 2026. Clarity added a robots.txt violations view on June 23, 2026, fed by Fastly, Amazon CloudFront and Cloudflare logs, and an AI scrape-to-referral ratio card in August 2026. IAB Tech Lab finalised version 2.3 of its podcast measurement guidelines on September 29, 2026, extending log-based counting to video podcasts. Cryptographic signing through Web Bot Auth may eventually reduce the reliance on IP checks, but as of October 2026 Google signed only a subset of Google-Agent requests, and Googlebot did not appear as a signer.

Timeline

  • Early 1990s: The NCSA HTTPd server introduces the Common Log Format.
  • 1995: The Apache HTTP Server project begins, inheriting the format; log analysers such as Analog appear.
  • March 1996: W3C working draft of the Extended Log File Format by Phillip Hallam-Baker and Brian Behlendorf.
  • 2005: Google acquires Urchin, later relaunched as the tag-based Google Analytics.
  • September 2016: IAB podcast measurement guidelines version 1.0, based on server logs.
  • October 19, 2016: Court of Justice of the EU rules in Breyer that dynamic IP addresses can be personal data.
  • December 2017: Podcast measurement guidelines version 2.0.
  • March 2025: Google begins daily refreshes of its crawler IP range files, according to its March 2026 documentation.
  • January 21, 2026: Microsoft Clarity launches Bot Activity using CDN log collection.
  • March 20, 2026: Google updates its crawler verification documentation; IP range files move to a new path.
  • June 23, 2026: Clarity adds robots.txt violation reporting from Fastly, CloudFront and Cloudflare logs.
  • July 1, 2026: Cloudflare launches crawl-to-referral reporting per AI operator.
  • September 29, 2026: IAB Tech Lab finalises podcast measurement guidelines version 2.3.
  • October 30, 2026: Cloudflare updates its Logpush documentation.

Summary

Who: Web servers and CDNs write the logs. Publishers, hosting platforms, SEO specialists, podcast measurement vendors and security teams analyse them, alongside tools such as Botify, Ahrefs, Microsoft Clarity and Cloudflare.

What: Server logs are line-by-line records of every request a server handles, typically in the Common, Combined or W3C Extended format, listing IP address, timestamp, request, status code, bytes, referrer and user agent.

When: The formats date from the early and mid 1990s. Logs were the main source of web analytics until tag-based tools took over in the 2000s, and returned to prominence with podcast measurement from 2016 and AI crawler analysis from 2024 onward.

Where: On origin servers and, for most commercial sites, at CDNs, from which they are exported to storage and analysis systems.

Why: They are the only record that captures every request, human or automated, including bots that never run analytics tags. Their weaknesses are self-reported user agents, personal-data obligations over IP addresses, and the fact that they record delivery rather than human attention.