Indexing is the stage of a search engine's pipeline where a fetched page is parsed, evaluated and stored in a structure that can be searched in milliseconds. Crawling brings bytes back from a web server. Indexing decides what those bytes mean, whether the document is worth keeping, and under which terms it can later be retrieved. A page that is crawled but never indexed cannot appear in any result, cannot be cited in an AI-generated answer, and to that engine does not exist.

The structure underneath

The data structure doing the work is the inverted index. A forward index maps each document to the words it contains, which is the wrong orientation for answering a query. An inverted index turns that around: every term points to a posting list of the documents containing it, with positions and frequencies attached. A two-word query becomes an intersection of two posting lists rather than a scan of the web. Pandu Nayak, then a Google vice-president of search, described the mechanism under oath in October 2023 as the equivalent of the index at the back of a book.

Google's account of what happens between fetch and storage is specific. After retrieval and rendering, the system parses the text along with key tags and attributes, works out whether the document duplicates something already known, groups similar pages into a cluster and picks one member as the canonical. The others become alternate versions served in particular contexts. Signals about the canonical are collected then, among them page language, the country the content is local to, and usability. The result is stored in what the documentation calls a large database hosted on thousands of computers, with a warning attached: indexing is not guaranteed, and not every page Google processes will be kept.

Scale explains the architecture. Completing its Caffeine indexing system on 8 June 2010, Google claimed results roughly 50% fresher than before, hundreds of thousands of pages processed in parallel every second, and a single database of close to 100 million gigabytes growing by hundreds of thousands of gigabytes daily. Cadence mattered more than size: the older design refreshed the index in layers on a batch schedule, so a page could wait weeks for its turn, while Caffeine analysed the web in small portions and updated continuously.

Canonical selection and rendering order

Clustering is where most indexing surprises originate, because a rel=canonical element is a hint rather than an instruction, weighed against internal links, sitemap entries, redirects, hreflang and other signals. Google stated in December 2025 that canonicalisation happens both before and after rendering, and advised against using scripts to rewrite a canonical to a different value from the one in the served HTML. The same updates carried a sharper warning: on encountering a noindex directive the system may skip rendering and JavaScript execution altogether, so a script written to remove that directive may never run.

Rendering sits between fetch and index for HTML pages, executing JavaScript in a headless browser so the indexer reads the rendered document object model. Google published a nine-point overview of its crawl logic on 3 March 2026 describing the schedule that feeds it. The fetch itself is handled by Googlebot, whose per-URL size limit fell from 15MB to 2MB in February 2026, and content past a truncation point is never rendered and therefore never indexed.

The controls

Two mechanisms are routinely confused. The robots exclusion protocol in robots.txt governs fetching. Robots meta tags and the X-Robots-Tag HTTP response header, carrying values such as noindex, govern indexing, and take effect only if the page can be fetched at all. John Mueller confirmed the consequence in September 2024: a URL blocked in robots.txt can still be indexed on the strength of inbound links, because the crawler cannot read the noindex it carries. Google added AI Mode to the robots meta tag documentation in March 2025, extending that page-level vocabulary to generative surfaces.

XML sitemaps remain the standard way to declare what exists. Microsoft's specification allows 50,000 URLs per sitemap file and 50,000 child files per index file, or 2.5 billion addresses through one entry point, figures published in August 2025alongside a requirement for ISO 8601 timestamps in the lastmod field. Gary Illyes described lastmod as a signal of site activity in May 2024, though not the sole determinant of crawl frequency.

Push-based alternatives are fragmented. IndexNow, launched by Microsoft with Yandex in October 2021, lets a site notify participating engines the moment a URL is added, changed or removed; by October 2023 it reported 60 million websites and 1.4 billion URLs submitted daily, with Microsoft issuing sector-specific guidance on submission cadence in September 2024. Google has never joined, an absence examined in December 2024 as a difference in philosophy rather than a technical obstacle, since participants must share submitted URLs with each other. Google's own Indexing API is narrow: documentation clarified on 4 September 2024 confirmed it covers job posting and broadcast event markup only, through publish and getMetadata endpoints, each with its own quota.

Origins

Early web search indexed metadata rather than text. ALIWEB, announced by Martijn Koster in November 1993, relied on site owners submitting their own descriptions. WebCrawler, built by Brian Pinkerton at the University of Washington and operational from April 1994, was the first crawler-based engine to index the full text of every page it fetched. Lycos went online that July with 54,000 documents. On 15 December 1995 Digital Equipment Corporation opened AltaVista, whose index covered roughly 16 million documents and set the benchmark for comprehensiveness until link-based ranking arrived with Google in 1998.

Later milestones concern freshness and format. Caffeine ended batch refreshes in 2010. Mobile-first indexing, announced in 2016, completed on 5 July 2024, after which pages unreachable on a mobile device dropped out.

Why the marketing community watches it

Indexation is the precondition for every organic impression and, increasingly, for citation inside AI answers. It is also the least controllable stage. Mueller told one site owner in July 2025 that minimal indexing on technically sound hostingusually means Google's systems are not convinced about the site overall, a framing that moves the problem from engineering to editorial judgement.

Mike Hardaker of Mountain Weekly News hit a daily quota on manual indexing requests after five submissions in April 2026 while updating 1,192 gear audits. Francois Zaninotto, chief executive of the studio Marmelab, disclosed on 21 May 2026 that search traffic had halved in six months as pages moved into the crawled-but-not-indexed bucket without explanation.

Limits and disputes

Coverage is selective and always has been. Nayak put the index at roughly 400 billion documents as of 2020, told the court that a bigger index is not necessarily better because it can be filled with junk, and indicated the figure had fallen for a period. Trillions of URLs have been crawled against that number.

The vocabulary used to report exclusion is contested. Google treats "crawled - currently not indexed" as a page it has read and declined to include, and Martin Splitt explained the related discovered-but-not-indexed status in an August 2024 video as a queueing outcome rather than an error. First-party data from an indexing monitoring vendor, cited in the Marmelab coverage, suggests 70% to 80% of pages carrying the first label had been indexed before, making it a record of removal rather than delay.

Measurement is fragile. The Search Console page indexing report stopped updating on 11 June 2026, with Mueller acknowledging the outage on 1 July and giving no restoration estimate. Faults have also produced false alarms, including an overnight episode in September 2026 in which indexed pages were briefly relabelled.

Two structural disputes sit above the technical ones. The first is coupling. One index feeds conventional results and generative answers, and a senior Google executive called separating the two a huge engineering project at a London conference on 11 February 2026, leaving publishers to choose between full participation and reduced visibility until regulators intervened. The second is ownership. On 2 September 2025 Judge Amit Mehta ordered Google to make search index and user-interaction data available to qualified competitors, treating the index as essential raw material for competition; Google appealed in January 2026.

Not the same as

Crawling is retrieval, and it succeeds or fails independently of what happens next.

Ranking operates after indexing, ordering documents already held for a specific query. A page can be indexed and never rank.

Rendering executes JavaScript to build the version of a page the indexer reads, and is skipped in some cases, including pages already marked noindex.

Grounding retrieves passages to support a generated answer. It draws on an index but optimises for different properties, and abstention is an acceptable outcome.

Recent developments

Microsoft set out the divergence on 6 May 2026, in a technical post explaining why an index built for AI grounding is not an index built for search. Its authors, Krishna Madhavan, Knut Risvik and Meenaz Merchant, argued that the unit of value shifts from the document to groundable information with clear provenance, that factual fidelity, source attribution, freshness and contradiction handling become primary quality measures, and that declining to answer is a valid outcome rather than a coverage failure.

Removal pressure has run alongside that. Reports of rising exclusion built from early April 2026, and by 14 May pages were vanishing from the index during heavy ranking volatility, with Mueller responding on 30 April that some sites go up and some go down. On 16 July 2026 Marie Haynes published an analysis arguing that competent but ordinary content is now losing its index place, putting the threshold for inclusion above technical correctness.

Timeline

  • November 1993: ALIWEB announced, indexing owner-submitted descriptions rather than page text
  • April 1994: WebCrawler becomes the first crawler-based engine to index the full text of pages
  • July 1994: Lycos launches with 54,000 indexed documents
  • 15 December 1995: AltaVista opens with an index of roughly 16 million documents
  • 1998: Google launches, combining an inverted index with link-based ranking
  • 8 June 2010: Caffeine completed, replacing batch refreshes with continuous indexing
  • 2016: Mobile-first indexing announced
  • October 2021: Microsoft and Yandex launch the IndexNow notification protocol
  • October 2023: Pandu Nayak testifies that the index held about 400 billion documents in 2020
  • 5 July 2024: Mobile-first indexing transition completed
  • 4 September 2024: Indexing API quota and eligibility documentation clarified
  • 2 September 2025: Judge Amit Mehta orders Google to share search index data with qualified competitors
  • 15 to 18 December 2025: Google documents noindex and canonical handling during JavaScript rendering
  • April 2026: Reports of rising crawled-not-indexed counts begin accumulating
  • 6 May 2026: Microsoft publishes its account of indexing for AI grounding
  • 11 June 2026: Search Console page indexing report stops updating
  • 16 July 2026: Marie Haynes publishes analysis of ordinary content losing index inclusion

Summary

Who: Search engine operators build and maintain indexes, with Google and Microsoft the two dominant western systems. Publishers, SEO practitioners and technical marketers manage eligibility through sitemaps, robots directives, canonical signals and submission protocols. Regulators and courts in the United States and Europe now treat index data as contested infrastructure.

What: The stage at which a fetched page is parsed, clustered against duplicates, assigned a canonical, scored for signals and stored in an inverted index, or dropped. Inclusion is selective and explicitly not guaranteed by Google's own documentation.

When: Full-text indexing dates to WebCrawler in April 1994, with continuous indexing arriving through Caffeine on 8 June 2010, mobile-first indexing completing on 5 July 2024, and AI grounding requirements documented from May 2026.

Where: In the operators' own infrastructure, described by Google as a database spread across thousands of computers, with publisher-facing reporting through Google Search Console and Bing Webmaster Tools.

Why: Nothing outside the index can be ranked, clicked or cited, which makes indexation the single dependency underneath organic search visibility and AI answer inclusion alike. That leverage is why exclusion patterns, reporting outages and court-ordered index sharing have all become live commercial issues.