Home › Guides › AI crawlers in your server logs
AI crawlers in your server logs
Direct answer: look for the named AI user-agents — GPTBot, ClaudeBot, PerplexityBot, CCBot, Meta-ExternalAgent, Bytespider, plus user-triggered fetchers like ChatGPT-User and Claude-User. The checker below counts their visits, error hits and top paths from your access log, locally.
Find the AI crawlers in this log
| Crawler | Visits | Errors | Top paths | First seen |
|---|
Two families, two meanings
Crawlers (GPTBot, ClaudeBot, PerplexityBot, CCBot) fetch systematically to build datasets or indexes — their visits are bulk events, often bursty, often from published IP ranges. User-triggered fetchers (ChatGPT-User, Claude-User, Perplexity-User) fetch a page because a person asked the assistant about it — their visits look like referrals and correlate with your content being cited. Lumping them together as "AI traffic" hides the difference between being scraped and being recommended.
What to look at in the results
Error hits per crawler: dataset-driven crawlers retry dead URLs, so one crawler with thousands of 404s is usually reading a stale list — check whether the paths match an old sitemap. Top paths: training crawlers go deep (archives, tag pages); search-oriented ones (OAI-SearchBot) behave more like a search engine. And any crawler claiming a big name from an unrelated network deserves the same reverse-DNS skepticism as a claimed Googlebot — see verifying crawler identity.
The robots.txt reality
You can disallow each named token in robots.txt, and the major operators document honouring it — that is the whole mechanism. It separates training crawlers from search crawlers usefully (blocking GPTBot does not block OAI-SearchBot unless you list it too; blocking Applebot-Extended opts out of AI training while Applebot search crawling continues). What robots.txt cannot do is enforce anything against actors who don't identify themselves honestly — for those, the evidence stays in logs like the one you just checked.
Frequently asked questions
Which AI crawlers should I expect to see?
The documented ones: GPTBot and OAI-SearchBot plus ChatGPT-User (user-triggered fetches) from OpenAI; ClaudeBot and Claude-User from Anthropic; PerplexityBot and Perplexity-User; CCBot (Common Crawl); Meta-ExternalAgent; Bytespider; Amazonbot; Applebot-Extended (Apple’s AI-training opt-out token, distinct from Applebot search crawling). Any of them can also be spoofed by unrelated scrapers — the UA is a claim.
Does blocking GPTBot keep my content out of AI training?
For compliant crawlers, a robots.txt disallow for GPTBot is the documented opt-out and the major operators state they honour it. It does nothing about non-compliant scrapers, content already collected, or user-triggered fetches (ChatGPT-User is a separate token with separate rules). Treat robots.txt as a policy signal to named operators, not a fence.
Why do AI crawlers hit errors so often?
They frequently crawl from URL lists (datasets, sitemaps, backlinks) rather than by following your live navigation, so stale and deleted URLs get retried long after a search engine would have given up. A high error share from one AI crawler usually points at your sitemap or at old URLs circulating in datasets — the top-paths column shows which.