![Complete Crawler List For AI User-Agents [Dec 2025]](https://i0.wp.com/cdn.searchenginejournal.com/wp-content/uploads/2025/11/ai-crawler-152.jpg?ssl=1 "Complete Crawler List For AI User-Agents [Dec 2025]")
AI visibility performs a vital function for SEOs, and this begins with controlling AI crawlers. If AI crawlers can’t entry your pages, you’re invisible to AI discovery engines.
On the flip aspect, unmonitored AI crawlers can overwhelm servers with extreme requests, inflicting crashes and surprising internet hosting payments.
Person-agent strings are important for controlling which AI crawlers can entry your web site, however official documentation is commonly outdated, incomplete, or lacking solely. So, we curated a verified checklist of AI crawlers from our precise server logs as a helpful reference.
Each user-agent is validated towards official IP lists when obtainable, making certain accuracy. We’ll keep and replace this checklist to catch new crawlers and adjustments to present ones.
The Full Verified AI Crawler Record (December 2025)
| Title | Goal | Crawl Fee of SEJ (pages/hour) | Verified IP Record | Robots.txt disallow | Full Person Agent |
|---|---|---|---|---|---|
| GPTBot | AI coaching knowledge assortment for GPT fashions (ChatGPT, GPT-4o) | 100 | Official IP Record | Person-agent: GPTBot Permit: / Disallow: /private-folder |
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; suitable; GPTBot/1.3; +https://openai.com/gptbot) |
| ChatGPT-Person | AI agent for real-time net shopping when customers work together with ChatGPT | 2400 | Official IP Record | Person-agent: ChatGPT-Person Permit: / Disallow: /private-folder |
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); suitable; ChatGPT-Person/1.0; +https://openai.com/bot |
| OAI-SearchBot | AI search indexing for ChatGPT search options (not for coaching) | 150 | Official IP Record | Person-agent: OAI-SearchBot Permit: / Disallow: /private-folder |
Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36; suitable; OAI-SearchBot/1.3; +https://openai.com/searchbot |
| ClaudeBot | AI coaching knowledge assortment for Claude fashions | 500 | Official IP Record | Person-agent: ClaudeBot Permit: / Disallow: /private-folder |
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; suitable; ClaudeBot/1.0; +claudebot@anthropic.com) |
| Claude-Person | AI agent for real-time net entry when Claude customers browse | <10 | Not obtainable | Person-agent: Claude-Person Disallow: /sample-folder |
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; suitable; Claude-Person/1.0; +Claude-Person@anthropic.com) |
| Claude-SearchBot | AI search indexing for Claude search capabilities | <10 | Not obtainable | Person-agent: Claude-SearchBot Permit: / Disallow: /private-folder |
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; suitable; Claude-SearchBot/1.0; +https://www.anthropic.com) |
| Google-CloudVertexBot | AI agent for Vertex AI Agent Builder (web site homeowners’ request solely) | <10 | Official IP Record | Person-agent: Google-CloudVertexBot Permit: / Disallow: /private-folder |
Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Construct/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/141.0.7390.122 Cell Safari/537.36 (suitable; Google-CloudVertexBot; +https://cloud.google.com/enterprise-search) |
| Google-Prolonged | Token controlling AI coaching utilization of Googlebot-crawled content material. | Person-agent: Google-Prolonged Permit: / Disallow: /private-folder |
|||
| Gemini-Deep-Analysis | AI analysis agent for Google Gemini’s Deep Analysis characteristic | <10 | Official IP Record | Person-agent: Gemini-Deep-Analysis Permit: / Disallow: /private-folder |
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; suitable; Gemini-Deep-Analysis; +https://gemini.google/overview/deep-research/) Chrome/135.0.0.0 Safari/537.36 |
| Gemini’s chat when a person asks to open a webpage | <10 | ||||
| Bingbot | Powers Bing Search and Bing Chat (Copilot) AI solutions | 1300 | Official IP Record | Person-agent: BingBot Permit: / Disallow: /private-folder |
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; suitable; bingbot/2.0; +http://www.bing.com/bingbot.htm) Chrome/116.0.1938.76 Safari/537.36 |
| Applebot-Prolonged | Doesn’t crawl however controls how Apple makes use of Applebot knowledge. | <10 | Official IP Record | Person-agent: Applebot-Prolonged Permit: / Disallow: /private-folder |
Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Model/17.4 Safari/605.1.15 (Applebot/0.1; +http://www.apple.com/go/applebot) |
| PerplexityBot | AI search indexing for Perplexity’s reply engine | 150 | Official IP Record | Person-agent: PerplexityBot Permit: / Disallow: /private-folder |
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; suitable; PerplexityBot/1.0; +https://perplexity.ai/perplexitybot) |
| Perplexity-Person | AI agent for real-time shopping when Perplexity customers request data | <10 | Official IP Record | Person-agent: Perplexity-Person Permit: / Disallow: /private-folder |
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; suitable; Perplexity-Person/1.0; +https://perplexity.ai/perplexity-user) |
| Meta-ExternalAgent | AI coaching knowledge assortment for Meta’s LLMs (Llama, and many others.) | 1100 | Not obtainable | Person-agent: meta-externalagent Permit: / Disallow: /private-folder |
meta-externalagent/1.1 (+https://builders.fb.com/docs/sharing/site owners/crawler) |
| Meta-WebIndexer | Used to enhance Meta AI search. | <10 | Not obtainable | Person-agent: Meta-WebIndexer Permit: / Disallow: /private-folder |
meta-webindexer/1.1 (+https://builders.fb.com/docs/sharing/site owners/crawler) |
| Bytespider | AI coaching knowledge for ByteDance’s LLMs for merchandise like TikTok | <10 | Not obtainable | Person-agent: Bytespider Permit: / Disallow: /private-folder |
Mozilla/5.0 (Linux; Android 5.0) AppleWebKit/537.36 (KHTML, like Gecko) Cell Safari/537.36 (suitable; Bytespider; https://zhanzhang.toutiao.com/) |
| Amazonbot | AI coaching for Alexa and different Amazon AI providers | 1050 | Not obtainable | Person-agent: Amazonbot Permit: / Disallow: /private-folder |
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; suitable; Amazonbot/0.1; +https://developer.amazon.com/help/amazonbot) Chrome/119.0.6045.214 Safari/537.36 |
| DuckAssistBot | AI search indexing for DuckDuckGo search engine | 20 | Official IP Record | Person-agent: DuckAssistBot Permit: / Disallow: /private-folder |
DuckAssistBot/1.2; (+http://duckduckgo.com/duckassistbot.html) |
| MistralAI-Person | Mistral’s real-time quotation fetcher for “Le Chat” assistant | <10 | Not obtainable | Person-agent: MistralAI-Person Permit: / Disallow: /private-folder |
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; suitable; MistralAI-Person/1.0; +https://docs.mistral.ai/robots) |
| Webz.io | Knowledge extraction and net scraping utilized by different AI coaching corporations. Previously often known as Omgili. | <10 | Not obtainable | Person-agent: webzio Permit: / Disallow: /private-folder |
webzio (+https://webz.io/bot.html) |
| Diffbot | Knowledge extraction and net scraping utilized by corporations all around the world. | <10 | Not obtainable | Person-agent: Diffbot Permit: / Disallow: /private-folder |
Mozilla/5.0 (Home windows; U; Home windows NT 5.1; en-US; rv:1.9.1.2) Gecko/20090729 Firefox/3.5.2 (.NET CLR 3.5.30729; Diffbot/0.1; +http://www.diffbot.com) |
| ICC-Crawler | AI and machine studying knowledge assortment | <10 | Not obtainable | Person-agent: ICC-Crawler Permit: / Disallow: /private-folder |
ICC-Crawler/3.0 (Mozilla-compatible; ; https://ucri.nict.go.jp/en/icccrawler.html) |
| CCBot | Open-source net archive used as coaching knowledge by a number of AI corporations | <10 | Official IP Record | Person-agent: CCBot Permit: / Disallow: /private-folder |
CCBot/2.0 (https://commoncrawl.org/faq/) |
The user-agent strings above have all been verified towards Search Engine Journal server logs.
Fashionable AI Agent Crawlers With Unidentifiable Person Agent
We’ve discovered that the next didn’t determine themselves:
- you.com.
- ChatGPT’s agent Operator.
- Bing’s Copilot chat.
- Grok.
- DeepSeek.
There is no such thing as a approach to monitor this crawler from accessing webpages apart from by figuring out the express IP.
We arrange a lure web page (e.g., /specific-page-for-you-com/) and used the on-page chat to immediate you.com to go to it, permitting us to find the corresponding go to document and IP handle in our server logs. Under is the screenshot:
What About Agentic AI Browsers?
Sadly, AI browsers resembling Comet or ChatGPT’s Atlas don’t differentiate themselves within the person agent string, and you may’t determine them in server logs and mix with regular customers’ visits.
That is disappointing for SEOs as a result of monitoring agentic browser visits to a web site is vital for reporting POV.
How To Examine What’s Crawling Your Server
Some internet hosting corporations provide a person interface (UI) that makes it simple to entry and take a look at server logs, relying on what internet hosting service you’re utilizing.
In case your internet hosting doesn’t provide this, you may get server log information (often situated /var/log/apache2/entry.log in Linux-based servers) through FTP or request it out of your server help to ship it to you.
After getting the log file, you possibly can view and analyze it in both Google Sheets (if the file is in CSV format), Screaming Frog’s log analyzer, or, in case your log file is lower than 100 MB, you possibly can attempt analyzing it with Gemini AI.
How To Confirm Legit Vs. Pretend Bots
Pretend crawlers can spoof respectable person brokers to bypass restrictions and scrape content material aggressively. For instance, anybody can impersonate ClaudeBot from their laptop computer and provoke crawl request from the terminal. In your server log, you will note it as Claudebot is crawling it:
curl -A 'Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; suitable; ClaudeBot/1.0; +claudebot@anthropic.com)' https://instance.comVerification might help to save lots of server bandwidth and forestall harvesting content material illegally. Probably the most dependable verification methodology you possibly can apply is checking the request IP.
Examine all IPs and scan to match if it’s one of many formally declared IPs listed above. In that case, you possibly can permit the request; in any other case, block.
Numerous forms of firewalls might help you with this through allowlist verified IPs (which permits respectable bot requests to go by), and all different requests impersonating AI crawlers of their person agent strings are blocked.
For instance, in WordPress, you need to use Wordfence free plugin to allowlist respectable IPs from the official lists (as above) and add blocking customized guidelines as beneath:
The allowlist rule is superior, and it’ll let respectable crawlers go by and block any impersonation request which comes from totally different IPs.
Nevertheless, please word that it’s doable to spoof an IP handle, and in that case, when bot person agent and IPs are spoofed, you received’t have the ability to block it.
Conclusion: Keep In Management Of AI Crawlers For Dependable AI Visibility
AI crawlers at the moment are a part of our net ecosystem, and the bots listed right here characterize the key AI platforms at the moment indexing the online, though this checklist is prone to develop.
Examine your server logs repeatedly to see what’s truly hitting your web site and be sure to inadvertently don’t block AI crawlers if visibility in AI engines like google is vital for your corporation. In case you don’t need AI crawlers to entry your content material, block them through robots.txt utilizing the user-agent title.
We’ll preserve this checklist up to date as new crawlers emerge and replace present ones, so we advocate you bookmark this URL, or revisit this text regularly to maintain your AI crawler checklist updated.
Extra Assets:
Featured Picture: BestForBest/Shutterstock
