A public question about a rarely discussed robots directive drew an unusually candid answer from a Google analyst, and exposed the gap between what the unavailable_after tag controls and what site owners want it to solve.

Gary Illyes, an analyst at Google, responded on LinkedIn to a technical question about the unavailable_after robots directive by conceding he did not know the answer, then offering a qualified view: pushing an expiry date further into the future is probably acceptable, but Google has to fetch the page again before it can register the change. The exchange was captured on July 28, 2026, roughly eighteen hours after the original post went up.

The question came from Javier Lorente Murillo, a freelance SEO consultant. It was addressed to the Google Search team and tagged three of its most visible members: John Mueller, Gary Illyes and Martin Splitt. By the time of capture the post had drawn 28 reactions and 5 comments.

The problem: 10,000 listings a month, most of them dead within three days

Lorente Murillo described a site with a structural mismatch between its stable and its transient inventory. According to his post, the site maintains a base of roughly 5,000 indexed URLs while generating around 10,000 new listings every month. Those listings have lifespans he put at 24 to 72 hours.

The ratio matters. Monthly new-listing volume runs at twice the size of the entire stable index, and almost all of that volume expires within three days. That produces a churn pattern in which the overwhelming majority of URLs Googlebot could discover on the site will be gone before any meaningful indexing cycle completes.

His stated aim in reaching for the unavailable_after directive was twofold: to prevent what he called massive 404 errors, and to manage crawl budget. The complication is renewals. Users on classifieds platforms routinely extend their ads, which means the site would need to push the expiry date forward dynamically on URLs that had already been crawled with an earlier date attached.

That produced the specific question. Would Googlebot treat a date that keeps sliding forward on the same URL as a signal to distrust, and eventually ignore, the tag altogether?

Illyes: "I have no idea, I have to check"

The answer was direct about its own limits. "that's a great question... I have no idea, I have to check," Illyes wrote.

He followed with a working assumption rather than documented behaviour. "my gut feeling is that it's fine to push forward the unavailable_after date BUT you need to keep in mind that we'll need to crawl the page again to 'see' the new date," he wrote. The comment continues beyond what the captured thread displays, and the remainder of the sentence is truncated in the source.

The operative constraint sits in that second clause. A directive embedded in a page's markup or HTTP headers is invisible to a search engine until the search engine fetches the page. An expiry date that changes on the fly therefore does not propagate on the fly. It propagates on Google's crawl schedule, which the site does not control.

For a page with a 72-hour lifespan, the practical question becomes whether a re-crawl happens inside that window at all. If it does not, the tag's value as a life-cycle management mechanism narrows considerably, because the date Google holds will frequently be the wrong one.

The HEAD request problem

A second Illyes comment, posted later in the thread, closed off a workaround that some site operators might otherwise assume. Replying to another participant, he wrote: "Andrew Glyntzos we rarely see HEAD requests so we'd still download the content."

The HTTP HEAD method retrieves response headers without the response body. In theory, a crawler could poll a URL cheaply with HEAD requests to check whether an X-Robots-Tag expiry header had changed, without paying the bandwidth cost of downloading the full document. Illyes's comment indicates that this is not how Google's fetching typically works in practice.

The consequence is arithmetic. If every check of a changed expiry date requires a full content download, the tag delivers no reduction in crawl cost per fetch. It can only reduce the number of URLs eligible for serving in search results, which is a different objective from reducing what Googlebot transfers.

That distinction has become more pointed as Google has tightened its own fetching constraints. PPC Land reported that Google documented a 2MB fetch limit for Googlebot in a Search Central blog post on March 31, 2026, authored by Illyes, alongside a relocation of crawler IP range files. Earlier the same month, Illyes and Martin Splitt explained that Googlebot is not a standalone program but one client of a central internal crawling platform shared across Google products.

The half the tag does not reach

Lorente Murillo returned to the thread with a follow-up that reframed the scope of the issue. "In any case, this would only solve 50% of the problem - specifically, users landing on 404 pages (as is currently the case, with error_404.html ranking in the top 10 most-viewed pages). We also see over 50% of Googlebot hits going to 404 pages, and I'm afraid the unavailable_after tag wouldn't fix that," he wrote.

Two figures in that comment are worth separating.

The first is that an error template file appears among the site's ten most-viewed pages. That is a measure of live human traffic arriving at dead ends, whether from search results, external links, bookmarks or internal navigation pointing at listings that have already expired.

The second is that more than half of Googlebot's requests to the site resolve to 404 responses. That is a crawl efficiency figure, and it describes a crawler spending the majority of its attention on URLs that no longer return content.

The unavailable_after directive addresses neither directly. It governs whether Google should stop showing a URL in search results after a given time. It does not change the HTTP status code the server returns when an expired listing is requested, and it does not remove links to expired listings from other sites or from the platform's own pages.

An alternative from the ecommerce side

Andrew Glyntzos, an SEO practitioner who identified himself in the thread as having two decades in the field, described handling a comparable pattern on a large ecommerce site where products and spare parts changed frequently.

According to his comment, the approach taken was an orphaned or archive treatment: keeping the page accessible, stating that the product was no longer available, and presenting a list of relevant alternatives. Some URLs were flagged as 404. He described the outcome as producing some SEO issues, but noted that conversions were higher than they had been when the site simply issued 301 redirects.

That is a trade acknowledged rather than resolved. Archive pages retain link equity and give arriving users a next step, at the cost of a growing set of indexable URLs describing inventory that cannot be bought.

What Google has previously said about 404s

The thread reopens a topic on which Google's public position has been consistent but frequently misread.

In June 2024, Illyes set out a two-part classification of 404 errors: URLs that should return a 200 status and are failing because of a technical fault, and URLs that were never intended to return content at all. The first category warrants repair. The second, in his framing, splits again into URLs that might still be useful to visitors and URLs that are irrelevant.

Two months later, in August 2024 SEO office hours, John Mueller and Martin Splitt addressed the ranking question directly, stating that 404 errors do not affect the rankings of other pages on a site. Mueller argued against redirecting deleted pages to loosely similar content or to a homepage, on the grounds that it degrades the visitor experience, and pointed instead toward informative 404 pages with redirects reserved for cases with a genuine replacement.

Google's documentation has also grown more specific about how error pages are processed. In December 2025, the company clarified that JavaScript rendering may be skipped for non-200 HTTP status codes, including 404, 301, 401 and 403 responses. The same update warned that pages carrying a noindex tag in their initial HTML may never have their JavaScript executed at all, which constrains any approach that tries to add or remove indexing directives client-side.

That last point has a bearing on the classifieds case. A platform that renders listing status through JavaScript, and expects Google to pick up a dynamically written expiry directive, is depending on a rendering step that may not run.

Why this matters beyond classifieds

High-turnover inventory is not confined to classified ads. Job boards, event listings, travel availability, ticketing, auction platforms and retail catalogues with rapid stock rotation all generate URLs whose useful life is measured in days or hours rather than months.

For the marketing and publishing side, the exchange carries three practical readings.

Crawl budget is not directly purchasable through markup. The unavailable_after directive operates on serving eligibility, and Illyes's comment about HEAD requests indicates that verifying a changed date still costs a full fetch. Sites hoping the tag will thin out crawler load are attaching that hope to a mechanism designed for a different purpose.

Documented directives are not necessarily well-instrumented. A Google analyst responding to a question about a supported robots directive with an admission that he would need to check is a data point about how much operational attention the tag receives. The directive exists in Google's specifications. Its behaviour under rapid, repeated modification appears to be undocumented territory.

Crawl waste has a compounding cost. Half of Googlebot's requests to a site resolving to errors is a figure with consequences beyond Google Search. PPC Land coverage of Google's crawling infrastructure has established that Googlebot operates as one consumer of a shared internal platform, meaning inefficiency at the fetch layer touches multiple Google surfaces. Separately, PPC Land has documented Cloudflare data showing AI crawl-to-referral ratiosranging from 118 crawls per human visitor to nearly 50,000, an indication of how much automated traffic now reaches commercial sites independent of search.

The thread has not produced a resolution. Illyes said he would check. What the exchange did produce is a clearer statement of the constraint: any expiry signal a site writes into a page is only as current as Google's most recent fetch of that page, and on inventory that expires in 24 to 72 hours, that is a demanding standard.

Timeline

Summary

Who: Gary Illyes, an analyst at Google and a member of the Search Relations team, responding to Javier Lorente Murillo, a freelance SEO consultant, in a public LinkedIn thread that also drew a comment from SEO practitioner Andrew Glyntzos. The post tagged John Mueller, Gary Illyes and Martin Splitt.

What: A question about whether repeatedly pushing the unavailable_after expiry date forward on the same URL carries negative SEO consequences. Illyes said he did not know and would need to check, offered a preliminary view that moving the date forward is probably acceptable, and stated that Google must crawl the page again to register the new date. In a follow-up comment he noted that Google rarely issues HEAD requests, meaning the content would still be downloaded in full. Lorente Murillo added that the tag would address only half of his problem, citing an error template ranking among his ten most-viewed pages and more than 50 percent of Googlebot hits landing on 404 responses.

When: The original post was published roughly eighteen hours before the thread was captured on July 28, 2026, with Illyes's first reply arriving about twelve hours before capture and his follow-up about three hours before.

Where: LinkedIn, in a public post directed at the Google Search team. The site under discussion is a classifieds platform with approximately 5,000 stable indexed URLs and around 10,000 new listings generated monthly, each with a lifespan of 24 to 72 hours.

Why: The exchange clarifies a limit on what the unavailable_after directive can do. Because the directive lives inside the page, a search engine only learns of a changed expiry date by fetching the page again, and because Google rarely uses HEAD requests, that fetch carries the full content download. Sites running high-turnover inventory, including job boards, event listings, ticketing and rapidly rotating retail catalogues, face the same structural gap between the crawl efficiency they want and the serving control the directive actually provides.