Bytespider is the web crawler operated by ByteDance, the Chinese company behind TikTok, Douyin, the Toutiao news aggregator and the Doubao assistant. It requests publicly reachable pages across the internet and returns their contents to ByteDance, where the data feeds search indexes, recommendation systems and the corpora used to train the company's large language models. The crawler exists for the same structural reason Googlebot exists: a company that wants its products to answer questions about the web first has to read the web. What separates Bytespider from other crawlers of comparable size is the volume of complaint attached to it, and an unresolved question about whether it honours the file publishers use to say no.

How Bytespider identifies itself

Crawlers announce themselves in the User-Agent header of every HTTP request. ByteDance's webmaster documentation, published on the Toutiao Search platform at zhanzhang.toutiao.com, lists three variants, one each for desktop, Android and iOS profiles. All three carry the token Bytespider with a capital B, and all three append a contact reference. Two contact strings circulate: the URL of the webmaster platform itself, and the address spider-feedback@bytedance.com. A representative desktop string reads Mozilla/5.0 (compatible; Bytespider; https://zhanzhang.toutiao.com/) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.0.0 Safari/537.36.

The mobile variants matter more than they look. Both declare Android 5.0 or iOS 7.1.2, operating system versions released in 2014, wrapped in build numbers no current handset reports. Log analysis filtering on browser version rather than on the Bytespider token will misclassify the traffic as obsolete mobile devices.

ByteDance also publishes a network footprint: ten address blocks in CIDR notation, each a /24 covering 256 addresses, for 2,560 in total. The list runs 110.249.201.0/24, 110.249.202.0/24, 111.225.148.0/24, 111.225.149.0/24, 220.243.135.0/24, 220.243.136.0/24, 220.243.188.0/24, 220.243.189.0/24, 60.8.123.0/24 and 60.8.151.0/24. Webmasters have found reverse DNS entries on those ranges in the form bytespider-[address].crawl.bytedance.com, a rough verification path. Rough, because ByteDance publishes no signed scheme comparable to the reverse-then-forward DNS lookup Google documents for Googlebot, and because observed traffic routinely originates outside the published blocks. The crawler-identification database at pdflibr.com records 2,501 distinct addresses under the Bytespider agent, and a further 1,492 under a separate fake Bytespider entry.

Where it sits in the crawl

The mechanics are unremarkable by design. Bytespider discovers URLs through links on pages it has already fetched, through XML sitemaps, and through seed lists assembled by ByteDance. It issues ordinary GET requests, follows redirects, and works through a site systematically rather than sampling it. The Toutiao webmaster platform offers a crawl frequency control, with ByteDance stating that changes take effect within one day, and an escalation address at zhanzhang@bytedance.com for sites experiencing slowdowns or outages.

Two properties distinguish a training crawler from a search crawler at the level of the transaction. A search crawler creates at least a theoretical return path: content is indexed, a result page is generated, a person clicks through. A training crawler consumes the page once and produces no citation and no referral. Bytespider straddles both categories, because ByteDance runs Toutiao Search alongside Doubao, but the second function dominates how the crawler is now classified.

Origin and evolution

Bytespider predates the generative AI era by several years. ByteDance launched Toutiao Search in August 2019 as a general web search portal attached to its news aggregator, and the crawler carrying the zhanzhang.toutiao.com reference was built to index for it. Webmaster forum reports place Bytespider hits in server logs from 2018, 2019, 2020 and 2022, arriving from the 220.243 and 111.225 ranges ByteDance later published.

That history sits awkwardly against the crawler's best-known origin story. When Fortune reported on Bytespider on October 3, 2024, drawing on research from bot management firm Kasada and the scraper monitor Dark Visitors, it described ByteDance as having released the crawler "sometime in April" of that year. Kasada chief executive Sam Crowther told Fortune that Bytespider was collecting at roughly 25 times the rate of OpenAI's GPTBot and 3,000 times the rate of Anthropic's ClaudeBot. The two accounts cannot both be right on the launch date. The most defensible reading is that a crawler built for search was rescaled for AI training across 2023 and 2024, and that April 2024 marks a change in behaviour rather than a first appearance.

The first substantial publisher confrontation came earlier. In December 2023, The Telegraph reported that UK news organisations including The Guardian and the Daily Mail had been targeted by Bytespider, with publishers raising copyright concerns and describing difficulty in blocking the bot. ByteDance told the newspaper the crawler served search optimisation purposes.

Measurement arrived in mid-2024. Cloudflare data published that July showed AI bots reaching roughly 39% of the top one million internet properties on its network in June 2024, with only 2.98% of those properties blocking or challenging the requests. Bytespider led every other AI bot on request volume and reached 40.40% of Cloudflare-protected properties, while being one of the crawlers least often named in robots.txt files, a gap between exposure and awareness that persisted for months.

Share then collapsed. Cloudflare figures covering January to July 2025 showed Bytespider falling from 14.1% of AI crawler traffic to 2.4% over the period, with no improvement in referral behaviour to accompany the retreat. Third-party readings of Cloudflare Radar data describe a sharp reversal in 2026: a climb from roughly 3.5% in early spring to 5.73% in April and 10.25% in May, taking fourth place among AI crawlers, followed by a fall back to 7.3% in June. Those figures come from independent analysts publishing monthly summaries of Radar data rather than from Cloudflare itself, and the volatility is itself the finding.

The robots.txt dispute

ByteDance's own documentation states that Bytespider follows the Robots Exclusion Protocol and instructs operators to block it with a two-line directive naming the user agent and disallowing the root path. Independent evidence points the other way often enough that the claim cannot be treated as settled.

Fortune's October 2024 report stated the crawler ignores robots.txt. TollBit research covering the first half of 2026, reported on August 14, 2026, found that Bytespider was one of three agents that reached disallowed pages on close to half of the European sites naming them in robots.txt, alongside ChatGPT-User and Youbot. Server administrators have documented individual incidents at scale, including a report of 1.4 million requests to a single site in one day consuming 14GB of transfer. Others report the opposite: that the crawler does stop on disallowed paths once it re-fetches an updated robots.txt file, with the delay between publication and compliance accounting for much of the apparent violation.

Spoofing complicates every measurement. Because ByteDance publishes no cryptographic verification method, any client can set the Bytespider token in its own header, and log-based studies counting violations by user agent alone will attribute impostor traffic to ByteDance. The counterargument is that a crawler operating from thousands of addresses outside its published ranges has made that defence hard to audit.

Why it matters for the marketing community

For publishers and advertisers the calculation is economic rather than technical. Bytespider consumes bandwidth and origin capacity while returning close to nothing in referral traffic. Cloudflare's crawl-to-refer ratio, which divides pages fetched by visitors returned, exists to quantify that asymmetry.

Blocking is not free either. A Rutgers Business School and Wharton School study, revised in April 2026, found that news publishers who blocked large language model crawlers through robots.txt lost roughly 7% of weekly traffic within six weeks, with the decline appearing in human browsing panel data. Earlier work on the same question put the figure at 23.1% of monthly visits with no corresponding reduction in AI citations. The two findings differ substantially in magnitude and have not been reconciled.

Awareness remains uneven. ProGEO.ai research published on March 31, 2026 found that 92.8% of Fortune 500 companies maintain a robots.txt file but only 11% name any specific AI user agent in it, and the default behaviour of the protocol treats silence as permission.

Bytespider and adjacent terms

TikTokSpider is a separate ByteDance user agent that crawls external content linked from TikTok and Douyin posts, narrower in scope than Bytespider's general sweep. Doubaobot is the agent tied specifically to the Doubao assistant. Directives naming one do not cover the others, and ByteDance operates several further agents beyond these three.

Baiduspider is frequently confused with Bytespider on the strength of the name and the shared country of origin. It is the indexing crawler for Baidu, a search engine, and blocking it removes a site from Baidu results. Blocking Bytespider carries no equivalent consequence in Google or Bing.

Common Crawl's CCBot collects pages into a public dataset that AI developers then draw on, making it an indirect training pipeline rather than a first-party one. The News/Media Alliance submitted a formal demand for content removal to the organisation on April 29, 2026.

Recent developments

Enforcement is shifting below robots.txt. Cloudflare published a policy update on July 1, 2026 replacing its earlier binary framing of AI bots with three behavioural categories, Search, Agent and Training, and set September 15, 2026 as the date from which Training and Agent crawlers will be blocked by default on advertising-carrying pages for domains newly joining its network. Search crawlers remain allowed under the same rule. Cloudflare on the same day abandoned per-crawl charging in favour of a model paying publishers when content contributes to a generated answer.

The industry has also begun sorting crawlers formally. IAB Australia guidance published in 2026 assigns every crawler to one of four verdicts and concludes that robots.txt is not a reliable control for live agents. Bytespider's position under such schemes remains contested, because it performs search indexing and training collection under a single name.

Timeline

  • 2018: Bytespider user agent first appears in webmaster server logs, arriving from ByteDance-owned address ranges
  • August 2019: ByteDance launches Toutiao Search, the web search portal the crawler was built to index for
  • January 2021: Crawler-identification database pdflibr.com records its first Bytespider observation
  • December 2023: The Telegraph reports UK publishers targeted by Bytespider; ByteDance describes the crawler as serving search optimisation
  • April 2024: Date given by Kasada and Dark Visitors for the crawler's release, disputed by earlier log evidence
  • June 2024: Cloudflare measures Bytespider reaching 40.40% of properties on its network, the highest of any AI bot
  • October 3, 2024: Fortune reports Kasada findings that Bytespider collects at 25 times the rate of GPTBot and 3,000 times that of ClaudeBot
  • January to July 2025: Bytespider share of AI crawler traffic falls from 14.1% to 2.4%
  • May 2026: Third-party analyses of Cloudflare Radar data place Bytespider at 10.25% of AI crawler traffic, fourth overall
  • June 2026: The same analyses record a fall to 7.3%
  • July 1, 2026: Cloudflare sets September 15, 2026 as the start of default blocking for Training and Agent crawlers on ad-bearing pages for new domains
  • August 14, 2026: TollBit research reports Bytespider among three agents reaching disallowed pages on close to half of European sites naming them

Summary

Who: ByteDance, the Beijing-based owner of TikTok, Douyin, Toutiao and the Doubao assistant, operates Bytespider. Publishers, ad-supported media businesses and infrastructure providers including Cloudflare, Kasada, Dark Visitors and TollBit have measured, documented and contested its behaviour.

What: A web crawler that fetches public pages and returns them to ByteDance for search indexing, recommendation systems and AI model training. It declares itself through three documented User-Agent variants carrying the Bytespider token, and operates from ten published /24 address blocks plus a much larger set of observed addresses.

When: Present in server logs from 2018 and tied to the August 2019 launch of Toutiao Search. Named in publisher complaints from December 2023, measured as the highest-volume AI bot in June 2024, reduced to 2.4% of AI crawler traffic by July 2025, and volatile through 2026 at between 3.5% and 10.25%.

Where: Across the public web, with disproportionate documented impact on news publishers in the United Kingdom, Europe and North America. Blocking guidance and enforcement now sit largely at the CDN and firewall layer rather than in robots.txt alone.

Why: ByteDance requires large volumes of text to compete in generative AI and to operate its search and recommendation products. Publishers face bandwidth and origin costs with no referral traffic in return, and no settled evidence on whether the crawler honours the exclusion directives they publish.