AI crawler

Also: GPTBot, ClaudeBot, PerplexityBot

An AI crawler is a bot that fetches web pages to feed AI systems — for training (GPTBot, ClaudeBot) or live answer retrieval (OAI-SearchBot, PerplexityBot). Blocking retrieval bots via robots.txt or bot protection removes content from AI search citations; blocking training bots only keeps it out of future models.

Reviewed June 2026

Why it matters

Many distributor storefronts run aggressive bot protection that blocks AI crawlers sitewide — so the catalog is invisible to ChatGPT and Perplexity no matter how good the data is. Auditing AI-crawler access is the first technical deliverable of any GEO engagement.

In practice

An industrial distributor running Cloudflare bot management noticed its hydraulics catalog had stopped appearing in ChatGPT and Perplexity answers. The WAF rule blocking all bots was also blocking OAI-SearchBot and PerplexityBot — the retrieval crawlers that fetch pages to ground cited answers. Cloudflare's 2025 data shows GPTBot, ClaudeBot, and CCBot receive the most full-disallow directives, so this misconfiguration is common; allowlisting the search/retrieval user-agents restored citation eligibility.

Related terms

15 minutes. The one constraint.

Book a 15-minute strategy call. No sales pitch — we’ll name the single constraint capping your growth right now and the one change with the highest payback.

Book a strategy call Or get the free written audit

AI crawler

Why it matters

In practice

llms.txt

Generative engine optimization (GEO)

15 minutes. The one constraint.