What is an AI crawler?
An AI crawler is an automated bot that collects web content to feed AI assistants. It works either for model training or for the real-time retrieval that backs answers with cited sources. The main ones are GPTBot and OAI-SearchBot (OpenAI), ClaudeBot (Anthropic), PerplexityBot, Google-Extended and CCBot (Common Crawl). None of them run JavaScript: they read only the page's initial HTML, which is why a JavaScript-rendered site looks empty to them.
Which are the main AI crawlers?
| CRAWLER | COMPANY | WHAT IT DOES |
|---|---|---|
| GPTBot | OpenAI | Training of the GPT models |
| OAI-SearchBot | OpenAI | ChatGPT search (answers with sources) |
| ClaudeBot | Anthropic | Claude training and retrieval |
| PerplexityBot | Perplexity | Its own index for cited answers |
| Google-Extended | Gemini training | |
| CCBot | Common Crawl | Open dataset used to train several models |
Source: each company's official documentation, July 2026.
Related entries
What crawlers can (or cannot) read defines a brand's AI Visibility; the article on why AI can't read PDFs shows those limits in practice. To receive crawlers properly, a site has to allow them in robots.txt and serve structured data, and a curated map in llms.txt finishes the job.
Frequently asked questions
Is an AI crawler the same as Google's bot?
The principle is the same, but Googlebot renders JavaScript and feeds a search index, while AI crawlers run no JavaScript and feed models and generated answers instead. A page can rank on Google and still be invisible to GPTBot.
How do I know whether AI crawlers visit my site?
Through your server logs: each crawler identifies itself in the user-agent (GPTBot, ClaudeBot, PerplexityBot and so on). If they never show up, check that robots.txt is not blocking them and that the site is indexed on Bing, ChatGPT's retrieval source.
Should I block or allow AI crawlers?
If you want to be cited in AI answers, allow them. Blocking GPTBot or PerplexityBot in robots.txt removes your brand from the pool of possible sources. Publishers that sell content block them; brands that want visibility allow them explicitly.