Home › SEO Bot-Log Checker

Bot-log checker: which crawlers visited, and what they hit

Direct answer: drop an access log below to see every recognised crawler — search engines, AI crawlers, SEO tools, social fetchers — with visit counts, error hits and top paths. Caution, in print: a user-agent string is a claim made by the visitor, not a verification. Confirm search-engine crawlers with reverse-DNS before acting on this list.

Check this access log for crawlers

Drop an access log here — or click to choose one. Nginx/Apache combined or common format, read in slices on this device; nothing is uploaded.

Crawler list last checked against the operators' published user-agent documentation: 2026-11-08. It is re-verified quarterly; a crawler launched after that date may be missing until the next pass.

What to do with the results

Error hits are the actionable column. A search bot spending visits on 404s is crawl budget burned and a sitemap or internal-link problem made visible — the paths it hits are in the table. Start with checking whether Googlebot is really crawling your site, including the reverse-DNS verification step. The newer question — which AI crawlers are in your server logs and what their visits mean — has its own guide. For the non-bot half of the file, Access-Log Triage covers statuses and error paths.

Frequently asked questions

Does a Googlebot user-agent prove Google visited?

No — and this page prints that caution because it matters. A user-agent is a string the visitor sends; anyone can send Googlebot’s. The checker reports what the log claims. Before acting on it (blocking, or celebrating a crawl surge), verify important crawlers with reverse-DNS: a real Googlebot IP resolves to a googlebot.com / google.com hostname and back. The guide linked below walks through it.

Which crawlers are recognised?

Search engines (Googlebot and its inspection/store variants, Bingbot, DuckDuckBot, YandexBot, Baiduspider, Applebot), AI crawlers (GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, PerplexityBot, CCBot, Meta-ExternalAgent, Bytespider, Amazonbot and others), SEO/measurement bots (AhrefsBot, SemrushBot, MJ12bot, DotBot), and social fetchers (facebookexternalhit, Twitterbot, LinkedInBot, Slackbot). The list is a dated asset — this page prints the date it was last checked, and unknown bot-like agents simply appear as regular traffic.

Why do bot error hits matter?

A crawler repeatedly hitting 404s wastes crawl budget and signals index problems: dead sitemap URLs, removed products still linked, redirect chains ending nowhere. A bot with a high error count and a short top-path list is usually pointing at one fixable thing.

My traffic tool says bots are 40% of visits. Will this match?

Probably not exactly: analytics tools classify with JavaScript beacons and fingerprinting (most bots never execute JS, so they are invisible there), while this checker reads the server log and classifies by user-agent string only. The log is the ground truth of requests made; treat percentages as "of logged requests", not "of audience".