Which AI crawlers are hitting your site? GPTBot, ClaudeBot and 22 more, by company and purpose

Last updated 10 November 2026.

The AI crawler list, by company and purpose

These are the 24 AI crawlers Lantern recognises. The name is the token to look for in a user agent, and to use in robots.txt.

The purpose matters more than the name:

• Training (12): collects pages to train models. It tells you a company has your content. It doesn't send visitors.
• AI search (5): builds the index an assistant searches when it answers. If these never read a page, the assistant can't cite it.
• Assistant (7): a live fetch, made because a person asked a question and the assistant needed your page to answer it. This is the closest thing to a ChatGPT "impression" you can measure.

Lantern also recognises the classic search crawlers (Googlebot, Bingbot, Applebot, YandexBot, DuckDuckBot, PetalBot) and SEO crawlers (AhrefsBot, SemrushBot, MJ12bot, DotBot), and keeps them apart from the AI ones.

Why your analytics doesn't show them

Google Analytics, Plausible, Fathom and most other tools count a visit when a browser runs their script. A crawler asks your server for the HTML and leaves. It never runs the script, so nothing is recorded. That is why GPTBot never shows up in GA4, however often it visits.

Checked against each vendor's own site on 10 November 2026.

Three ways to see which AI crawlers read your site

1. Search your server logs

If you have access logs, one command gives a rough count per crawler:

grep -oE "GPTBot|OAI-SearchBot|ChatGPT-User|ClaudeBot|Claude-User|PerplexityBot|Perplexity-User|Meta-ExternalAgent|Bytespider|Amazonbot|CCBot" access.log \
  | sort | uniq -c | sort -rn

It works, but only where you have logs. Vercel, Netlify and most serverless hosts keep them for a short time or not at all, and a count doesn't tell you which pages were read.

2. Use your CDN

If your site is behind Cloudflare, its dashboard reports AI crawler requests and lets you block them. It shows requests, not what happened next: you can't see whether the assistant then sent you a visitor.

3. Report them to your analytics

Lantern takes a few lines on your server: a ready-made proxy.ts for Next.js, or one HTTP endpoint for any other server. From then on the dashboard shows every crawler by company and purpose, when it last came, and which pages it read. Setup is in the docs.

From crawler to visitor to customer

A crawler visit is the first step of three, and Lantern shows all of them on one dashboard:

1. The crawler reads your page. Shown by company, purpose and page.
2. The assistant sends you a person. Visitors from ChatGPT, Perplexity, Claude and Gemini land in their own AI channel, not in Referral or Direct.
3. That person pays. Connect a payment provider and the AI channel gets a revenue column. See revenue analytics.

You can ask about any of it from an assistant, too. Lantern's MCP server has a tool for AI crawlers, so Claude or Cursor can answer "which AI crawlers read my docs this week?".

Allowing or blocking AI crawlers in robots.txt

Use the tokens from the list above. A common setup lets the search and assistant crawlers in, so assistants can cite you, and keeps training crawlers out:

# Let AI search and assistants read the site
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: PerplexityBot
User-agent: Claude-SearchBot
User-agent: Claude-User
Allow: /

# Keep training crawlers out
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: CCBot
User-agent: Bytespider
Disallow: /

robots.txt is a request, not a lock: well-behaved crawlers follow it and some don't. Watching what really arrives is the only way to know.

Questions