AI Crawler Checker

An AI crawler is a bot operated by an AI company that reads web pages so a model can use them. They fall into three groups that people routinely confuse. Training crawlers such as GPTBot, ClaudeBot and Google-Extended collect text that may be used to train future models. Search crawlers such as OAI-SearchBot, PerplexityBot and Claude-SearchBot build the index an assistant queries when it answers. Live crawlers such as ChatGPT-User and Perplexity-User fetch a page in the moment, because a user just asked something that requires it. The distinction matters because the three groups have different consequences. Blocking a training crawler affects models that do not exist yet. Blocking a search or live crawler removes you from answers being generated today.

It depends on what you sell. If your revenue comes from people reading your pages, as with a news site or an ad-funded publisher, blocking training crawlers is a defensible position: you are refusing to supply raw material for free. Several large publishers have taken exactly that route. If you want to be recommended by assistants, blocking is self-defeating, and blocking the wrong crawler is worse than blocking none. A site that blocks OAI-SearchBot cannot appear in ChatGPT answers, no matter how good its content is. The useful position for most businesses is to allow search and live crawlers, then decide separately about training crawlers, which is a licensing question rather than a visibility one.

Because a single verdict of allowed or blocked hides the decision that actually matters. Most robots.txt advice online treats all AI bots as one category, which leads site owners to copy a blanket block and then wonder why they disappeared from assistant answers. OpenAI alone operates three distinct agents with three distinct purposes, and they are configured independently. You can refuse to feed training while remaining fully visible in search, and that is the configuration most businesses actually want. The tool labels every crawler by its effect so the trade-off is explicit: refusing training data costs you nothing in visibility, refusing the search index costs you the answers.

Everything is allowed. The absence of a robots.txt file is not a neutral state that crawlers interpret cautiously: it is an implicit permission to crawl the entire site. This tool reports that case explicitly rather than showing an error, because it is the situation of most sites and it is a valid configuration. That said, having no robots.txt means you also have no place to declare your sitemap, and no way to change your mind later without touching your server. Publishing a minimal file that allows what you want and declares your sitemap costs nothing and gives you a control surface you will eventually need.

No, and this is the most common misunderstanding about the whole subject. Allowing crawlers is a permission, not a recommendation. It removes an obstacle between your content and the model, but the model still decides whether your page deserves to be quoted when someone asks a question. Citation depends on whether your content answers the question specifically, whether it carries verifiable facts, and whether your domain carries enough authority for the engine to trust it over an alternative. Access control is the floor, not the ceiling. The sites cited most often in AI answers are not the ones with the most permissive robots.txt, they are the ones with the most quotable content.

It applies the standard robots.txt matching rules rather than a simplified reading. First it selects the group that applies to each crawler: an exact user-agent match wins, and the wildcard group is used only when no named group exists for that bot. Then, among the directives in that group, the one whose path pattern is longest wins, and at equal length Allow beats Disallow. The tool also supports the two wildcard characters the specification defines, asterisk for any sequence and dollar for end of URL. This matters because a site with several overlapping rules can be allowed and blocked at once depending on which directive a crawler applies, and a naive checker will give you the wrong answer.