AI Crawlers Explained: GPTBot, ClaudeBot, PerplexityBot

Between 2023 and 2026, at least 9 major AI companies deployed dedicated web crawlers: OpenAI launched GPTBot and OAI-SearchBot, Anthropic launched ClaudeBot, Perplexity AI launched PerplexityBot, Google introduced Google-Extended, ByteDance released Bytespider, Apple added Applebot-Extended, Meta deployed Meta-ExternalAgent, and Common Crawl continued operating CCBot as the backbone dataset for many AI training pipelines. Every site owner now faces a decision they did not face in 2022: which of these crawlers to allow, which to block, and how to configure robots.txt to communicate that decision precisely.

This is not a minor technical footnote. These crawlers determine whether content appears in AI-generated answers from ChatGPT Search, Perplexity, Google AI Overviews, and Apple Intelligence. Understanding how ChatGPT chooses sources to cite starts with understanding which crawlers you let in.

Why AI Crawlers Are Now a First-Class SEO Concern

AI crawlers determine AI search visibility. Traditional search crawlers determine Google and Bing rankings. These are separate systems.

Googlebot, Bingbot, and their variants have been the dominant crawl concerns for two decades. The logic was simple: if Googlebot can access and index a page, it can rank. Block Googlebot, and the page disappears from search. Everything else was secondary.

That logic holds for traditional search. It does not extend to AI search. When a user asks ChatGPT Search a question, the answer draws on content OpenAI’s crawlers have accessed (via OAI-SearchBot) combined with Bing index data. When Perplexity synthesizes an answer, it draws on real-time crawl results from PerplexityBot. Google AI Overviews use Googlebot data but Gemini training uses Google-Extended.

Each AI system has its own crawler. Each crawler can be allowed or blocked independently via robots.txt. And all major AI crawlers publicly committed to honoring robots.txt directives as of 2024. This means site owners have genuine, enforceable control.

The problem is that most sites have not updated their robots.txt to address any of these crawlers. A default WordPress, Webflow, or Squarespace install from 2022 has no instructions for GPTBot, ClaudeBot, or PerplexityBot. The crawlers default to treating that as an open invitation.

The Major AI Crawlers: A Complete Reference

The table below covers every major AI crawler active as of May 2026. User-agent strings are case-sensitive in robots.txt directives.

Crawler	Company	User-Agent String	Primary Purpose	Official Docs
GPTBot	OpenAI	`GPTBot`	Training data	platform.openai.com/docs/gptbot
OAI-SearchBot	OpenAI	`OAI-SearchBot`	ChatGPT Search real-time results	platform.openai.com/docs/plugins/bot
ClaudeBot	Anthropic	`ClaudeBot`	Training + research features	anthropic.com/crawling-policy
Claude-Web	Anthropic	`Claude-Web`	Legacy (older Anthropic crawler)	Same policy
PerplexityBot	Perplexity AI	`PerplexityBot`	Real-time search results	docs.perplexity.ai/docs/perplexitybot
Google-Extended	Google	`Google-Extended`	Gemini training data	developers.google.com/search/docs/crawling-indexing/overview-google-crawlers
Bytespider	ByteDance	`Bytespider`	TikTok, Doubao AI training	No official docs (as of May 2026)
CCBot	Common Crawl	`CCBot`	Open training dataset	commoncrawl.org/ccbot
Applebot-Extended	Apple	`Applebot-Extended`	Apple Intelligence training	support.apple.com/en-us/111900
Meta-ExternalAgent	Meta	`meta-externalagent`	Meta AI training	developers.facebook.com/docs/sharing/bot

GPTBot vs OAI-SearchBot: The Distinction Most Sites Miss

GPTBot and OAI-SearchBot are separate crawlers with separate purposes. GPTBot collects training data for future OpenAI model versions. OAI-SearchBot powers ChatGPT Search, providing real-time web results to users asking questions right now.

Blocking GPTBot removes a site from OpenAI’s training pipeline. It does nothing to ChatGPT Search visibility. Blocking OAI-SearchBot removes a site from real-time ChatGPT Search results, regardless of what GPTBot has previously indexed. Most articles on this topic treat them as the same crawler. They are not.

A publisher that blocks GPTBot (to protect content from training use) but allows OAI-SearchBot will still appear in ChatGPT Search citations. This is the correct configuration for many commercial and media sites.

Google-Extended vs Googlebot

Blocking Google-Extended does not affect Google Search rankings. Google confirmed this explicitly in documentation published in 2023. Google-Extended controls whether content feeds into Gemini’s training data and, by extension, Google AI Overviews. Googlebot controls organic search indexing. They are separate directives and behave independently.

PerplexityBot: 2024 Compliance Controversy, Resolved

In mid-2024, Wired and other publications reported that PerplexityBot was crawling pages marked as disallowed in robots.txt. Perplexity AI disputed some of the characterizations and updated its crawl compliance practices. As of Q4 2024, PerplexityBot behavior aligns with published robots.txt directives. Sites that block PerplexityBot opt out of Perplexity citation; sites that allow it are eligible for attribution in Perplexity answers.

Bytespider: High Crawl Volume, No Official Docs

Bytespider is the ByteDance crawler used for training TikTok-related AI products and Doubao (ByteDance’s LLM). It operates at unusually high crawl frequencies reported by server log analysis across hosting forums. Unlike OpenAI, Anthropic, and Google, ByteDance has not published official crawler documentation as of May 2026. Sites with tight crawl budgets or API rate limits frequently block it preemptively.

CCBot: The Invisible Training Layer

Common Crawl is a nonprofit that maintains an open web dataset crawled by CCBot. This dataset underlies training for many AI models beyond just Common Crawl’s own projects. LLaMA (Meta), Falcon, and numerous open-source models were trained partly on Common Crawl data. Blocking CCBot does not block any specific AI product, but it opts a site out of the broad open AI training ecosystem.

How to Control AI Crawler Access via robots.txt

The robots.txt standard (RFC 9309, finalized September 2022) applies to all crawlers that commit to honoring it. All crawlers listed in this article have publicly committed to robots.txt compliance.

Robots.txt files live at the root of a domain: https://yourdomain.com/robots.txt. Changes typically take effect within 24 to 48 hours as crawlers re-read the file on their standard refresh cycle.

Block a specific AI crawler entirely

User-agent: GPTBot
Disallow: /

Allow a specific AI crawler to crawl everything

User-agent: GPTBot
Allow: /

This is also the default if no directive exists, so it only needs to be stated explicitly if an earlier wildcard blocks all crawlers.

Block all AI crawlers with wildcard, then allow specific ones

User-agent: *
Disallow: /

User-agent: Googlebot
Allow: /

User-agent: Bingbot
Allow: /

User-agent: OAI-SearchBot
Allow: /

User-agent: PerplexityBot
Allow: /

This approach blocks everything by default and explicitly allows only approved crawlers. Note: the wildcard User-agent: * does not match Googlebot if Googlebot has its own explicit block. Explicit rules for a specific user-agent override wildcard rules for that agent.

Block training crawlers, allow search crawlers (the most common nuanced config)

This configuration protects content from being used as training data while maintaining AI search visibility:

# Block OpenAI training crawler
User-agent: GPTBot
Disallow: /

# Allow OpenAI search crawler (ChatGPT Search)
User-agent: OAI-SearchBot
Allow: /

# Block Anthropic training crawler
User-agent: ClaudeBot
Disallow: /

# Block Google Gemini training (does not affect Google Search rankings)
User-agent: Google-Extended
Disallow: /

# Allow Perplexity search crawler
User-agent: PerplexityBot
Allow: /

# Block high-volume ByteDance crawler
User-agent: Bytespider
Disallow: /

Block AI crawlers from specific sections only

A paywall or subscription site may want AI crawlers to see public pages (for brand visibility) but not gated content:

User-agent: GPTBot
Disallow: /members/
Disallow: /premium/
Disallow: /subscriber-content/
Allow: /

User-agent: ClaudeBot
Disallow: /members/
Disallow: /premium/
Disallow: /subscriber-content/
Allow: /

What robots.txt cannot do

Robots.txt blocks crawl access. It does not remove content from an AI model’s existing training data if that content was crawled before the disallow directive was added. OpenAI offers a separate opt-out form for removal from training data at platform.openai.com. Google and Anthropic have analogous requests processes.

The Strategic Question: Block or Allow?

Whether to allow AI crawlers depends on the business model, content type, and visibility goals. There is no universal answer.

Arguments for allowing search-focused AI crawlers (OAI-SearchBot, PerplexityBot):

Informational content that earns brand mentions in AI answers reaches users who never visit the site directly. A Perplexity citation for a product comparison generates awareness even with zero click-through.
Content blocked from AI search is invisible to a growing query segment. By 2025, ChatGPT reached 500 million weekly users (OpenAI, late 2024). Perplexity reached 100 million monthly users (Perplexity investor disclosure, early 2025). These are not niche audiences.
AI citations reinforce topical authority signals that may feed back into traditional SEO rankings through brand search volume and link acquisition.

Arguments for blocking training crawlers (GPTBot, ClaudeBot, CCBot):

Content-as-product businesses: publishers, data providers, and research organizations whose value proposition is the content itself have legitimate grounds to block training use. Using their content to train a competitor’s AI product is a direct business harm.
Legal uncertainty: as of 2026, no U.S. court has issued a definitive ruling on whether training AI on web content without permission constitutes copyright infringement. The New York Times v. OpenAI case (filed December 2023) remains active. Blocking training crawlers reduces exposure while legal frameworks develop.
Crawl budget: high-volume crawlers like Bytespider consume server resources and can impact crawl budget for smaller sites on shared hosting.

Practical guidance by site type:

Marketing and informational sites: allow OAI-SearchBot, PerplexityBot, and Applebot-Extended. Block Bytespider if server resources are constrained. Allow or block training crawlers based on content exclusivity.
E-commerce sites: allow all search-focused crawlers. Training crawlers are lower priority but unlikely to cause harm.
Subscription publishers and paywalled content: block training crawlers entirely, allow search crawlers with path restrictions to keep public content visible.
Large editorial sites with API rate limits: block Bytespider. Evaluate all others per editorial policy.

Rate Limiting and Monitoring AI Crawler Activity

How do you know which AI crawlers are actually visiting? Server logs contain the answer.

Every HTTP request to a server includes a User-Agent header. AI crawlers identify themselves using the strings listed in the reference table above. Filtering server logs by these strings shows crawl frequency, volume, and which pages they prioritize.

Basic log analysis using grep (for Apache or Nginx access logs):

grep "GPTBot" /var/log/nginx/access.log | wc -l
grep "ClaudeBot" /var/log/nginx/access.log | wc -l
grep "PerplexityBot" /var/log/nginx/access.log | wc -l
grep "Bytespider" /var/log/nginx/access.log | wc -l

For sites on Cloudflare, Firewall Analytics shows bot traffic by user-agent string without requiring server log access. Cloudflare also allows rate-limiting rules targeting specific user-agents, which is the practical approach for managing Bytespider volume.

The Crawl-delay directive in robots.txt instructs crawlers to wait a specified number of seconds between requests:

User-agent: Bytespider
Crawl-delay: 10

Compliance with Crawl-delay varies. Googlebot honors it. OpenAI’s crawlers honor it. Bytespider’s compliance is inconsistent based on reports from the webmaster community in 2024 and 2025. For crawlers with poor Crawl-delay compliance, a Cloudflare firewall rule targeting the user-agent string is more reliable.

Signs of unusual AI crawler activity:

Crawl frequency exceeding 1 request per second from a single IP range
Repeated requests for the same URL within minutes
Crawl spikes on new pages published within 24 to 48 hours (consistent with real-time search crawlers like OAI-SearchBot and PerplexityBot)
High 404 rates from a specific user-agent (common with older CCBot versions)

How AI Crawlers Connect to an SEO Audit

Standard SEO audits check Googlebot accessibility. A complete 2026 SEO audit checks AI crawler accessibility separately.

A Screaming Frog crawl, a Semrush site audit, or a manual robots.txt review optimized for Google does not surface AI crawler configuration issues. The checklist is different:

Is robots.txt present and valid (parseable per RFC 9309)?
Which AI crawlers are explicitly addressed in the robots.txt file?
Are training crawlers and search crawlers differentiated (GPTBot vs OAI-SearchBot)?
Are high-volume crawlers like Bytespider handled separately?
Does the site have an llms.txt file providing structured guidance to AI systems? (See the full explanation in what is llms.txt.)
Is structured data present to help AI systems understand page context?
Are canonical signals consistent to prevent AI systems from indexing duplicate content?

The relationship between AI crawler configuration and overall AI search visibility is explained in more depth in GEO vs SEO: What’s Different in 2026.

SEO Audit MCP’s GEO agent checks robots.txt configuration for AI crawlers as part of its standard audit output, surfacing which crawlers are explicitly addressed, which are missing from the directive set, and whether the training vs search distinction is correctly handled.

Frequently Asked Questions

Does blocking AI crawlers hurt traditional SEO rankings?

No. Blocking GPTBot, ClaudeBot, CCBot, or any AI-specific crawler has no effect on Googlebot or Bingbot indexing. These are separate user-agents. A Disallow: / for GPTBot does not instruct Googlebot. Rankings in Google and Bing are unaffected.

The one exception: blocking Google-Extended can reduce Google AI Overviews visibility, but Google explicitly confirmed it does not affect organic Google Search rankings. The directive only governs Gemini training data.

Are AI companies legally required to honor robots.txt?

No legal obligation exists in most jurisdictions as of 2026. The robots.txt standard is a voluntary protocol. However, OpenAI, Anthropic, Google, Perplexity, Apple, Meta, and Common Crawl have all publicly committed to honoring it. Violation of these public commitments creates legal exposure in other areas (terms of service, CFAA claims), which creates a practical incentive for compliance even without direct legal requirements.

How do I see which AI crawlers are currently visiting my site?

Check server access logs filtered by user-agent string. On Apache or Nginx, grep for each crawler name listed in the reference table. On Cloudflare, use Firewall Analytics under Security. On Cloudflare Radar, the Bot Score analytics tab shows bot traffic by category. For sites using Google Search Console, the crawl stats report shows Googlebot and Google-Extended but not third-party AI crawlers.

Should a site add an llms.txt file?

Yes, with low effort and no downside. llms.txt is an emerging standard (proposed by fast.ai in 2024) that provides AI systems with structured information about a site: purpose, key pages, what to include or exclude from AI-generated answers. The standard is not yet universal, but Perplexity and some Anthropic tools already reference it. A basic llms.txt file takes under 30 minutes to create and serves as a structured signal for AI systems that encounter it. What Is llms.txt covers the format and implementation in full.

What about crawlers not listed here?

The AI crawler landscape is active. New crawlers emerge as AI products proliferate. The practical approach: monitor server logs for unknown user-agents requesting pages at bot-like frequency, look up the IP range against published crawler documentation, and add directives as needed. OpenAI, Anthropic, and Perplexity all publish their crawler IP ranges alongside their official documentation.