Should You Block AI Crawlers on Your Server? A Decision Framework for AI Visibility

Summary

Blocking every AI crawler protects content from AI training but removes your site from AI-generated answers and citations, which is a real cost for most businesses. Most AI providers now separate training crawlers (like GPTBot) from search or retrieval crawlers (like OAI-SearchBot), so you can often block training while staying visible in AI search. The right answer depends on whether your content’s value comes from scarcity or from being found.

A hosting customer messages you after noticing GPTBot and ClaudeBot showing up hundreds of times a week in their access logs. They want to know if these bots should be blocked, and they want an answer today, not a philosophy lecture. The honest answer is that “block all AI bots” and “allow all AI bots” are both bad defaults. Some AI crawlers exist to train models on your content. Others exist to answer a user’s question right now, using your page as the source and, in some cases, sending them a link back to you. Treating them the same throws away the second kind of value to guard against the first.

What is an AI crawler, and how is it different from a search crawler?

An AI crawler is an automated bot, identified by a user-agent string, that fetches web pages on behalf of an AI company. Some AI crawlers exist to collect training data for a future model. Others exist to retrieve current information at the moment a user asks a question inside a chat or search product.

That distinction matters more than the word “AI” in the name. A training crawler, like OpenAI’s GPTBot or Anthropic’s ClaudeBot, ingests your page as one data point among billions, with no promise that your specific page ever gets cited or credited. A search or retrieval crawler, like OpenAI’s OAI-SearchBot or Anthropic’s Claude-SearchBot, fetches a page specifically so an AI product can answer a live question, and it typically does so with attribution back to the source.

Google runs a related but structurally different setup. Google-Extended is not a separate bot with its own traffic pattern, it is a control token inside robots.txt that Googlebot itself checks to decide whether a page can be used to train Gemini.

A fourth category worth knowing: user-triggered fetchers, like ChatGPT-User, which visit a page because a person asked an AI assistant to open or summarize it in real time. These behave more like a single visitor than a crawler, and providers note that standard robots.txt disallow rules may not apply to them in the same way.

Crawler type Purpose Example bots Typical effect of blocking
Training crawler Collects content for model training, no live citation GPTBot, ClaudeBot Content excluded from future model training; no immediate visibility loss
Search / retrieval crawler Fetches content to answer a live user question OAI-SearchBot, Claude-SearchBot, PerplexityBot Site becomes ineligible for citation in that product’s AI answers
Training signal (via existing crawler) Flags content for training without a separate bot Google-Extended (read by Googlebot) Excluded from Gemini training; does not affect Google Search ranking
User-triggered fetch Retrieves a specific page a person asked the assistant to open ChatGPT-User May not be reliably stoppable by robots.txt alone

What actually happens when you block an AI crawler?

Blocking an AI crawler through robots.txt is a request, not a lock. The file is a plain-text standard, formalized as RFC 9309, and it works because well-behaved crawlers choose to honor it, there is no technical mechanism in robots.txt itself that prevents a bot from ignoring the rule.

For the major AI companies’ named crawlers, that voluntary compliance generally holds: a Disallow rule for GPTBot in your robots.txt file stops GPTBot from fetching those paths. What it does not do is stop content that was scraped before you added the rule, content mirrored on other sites, or content picked up through datasets like Common Crawl that some models draw on independently of a direct crawl.

If a server administrator wants a harder guarantee than voluntary compliance, the next layer is server-side enforcement: firewall rules, rate limiting, or a security module that blocks requests by user-agent or IP range regardless of what robots.txt says. Plesk’s Security Core, which includes ModSecurity rules, can be configured to enforce this kind of blocking at the web server level rather than relying on the crawler to comply.

What do you gain and lose by blocking AI crawlers?

The case for blocking centers on control: if your business model depends on content scarcity, a paywalled research report, a proprietary dataset, or licensed media, then every crawl is a potential competitor input, and blocking training crawlers is a reasonable default.

The case against blocking centers on visibility: AI assistants and AI-powered search features increasingly answer questions that used to send a click to a website. If your crawler rules block the retrieval bots along with the training bots, your content simply cannot be the source of that answer, no matter how good it is. For most content businesses, that is a bigger loss than the training exposure it prevents.

A middle position is now common and, for most sites, the more defensible one: block the training-only crawlers (GPTBot, ClaudeBot, Google-Extended) while allowing the search and retrieval crawlers (OAI-SearchBot, Claude-SearchBot, PerplexityBot) to keep the door open for citation without agreeing to unrestricted training use.

How should you decide, site by site or server by server?

  1. Identify what the content actually is. Public marketing pages, documentation, and blog posts generally benefit from AI visibility. Client portals, paywalled content, and regulated or contractual data generally do not belong in front of any crawler, AI or otherwise.
  2. Separate training crawlers from search crawlers in your rules rather than writing one blanket Disallow line for “AI.”
  3. Check your Log File Analyzer or access logs for which bots are actually visiting before writing rules for bots that never show up. Plesk’s SEO Toolkit includes a Log File Analyzer that reports user-agents and crawl volume by bot, which turns this from a guess into a five-minute check.
  4. Decide per subscription, not just per server, if you host multiple client sites. An agency client selling proprietary courses has different needs than one running a public support wiki.
  5. Write the robots.txt rules and, where a harder guarantee is required, add server-level rules through Security Core.
  6. Revisit the list periodically. Crawler user-agent strings change, and new ones appear, so a robots.txt file written a year ago may already be missing bots that did not exist then.
  7. Document the decision and the reasoning, especially for agencies making this call on a client’s behalf, so it can be revisited when the client’s content strategy changes.

How do you apply the decision in Plesk?

Once you know which bots to allow and which to block, the rule itself is a short robots.txt edit, and Plesk gives you two practical paths to apply it. For a single site, the file can be edited directly through Plesk’s file manager under the domain’s document root. For a hosting business managing this decision across many customer sites with the same policy, the same rule set can be templated and rolled out per subscription rather than edited one site at a time.

For anything beyond “trust the crawler to read the file,” blocking has to happen at the server layer. This is where Security Core’s ModSecurity rules come in: they can inspect the user-agent header on every request and reject it before the page is served, which holds even against a bot that does not honor robots.txt.

Blocking AI crawlers is not a single decision, it’s a set of decisions, one per crawler type, and the right default in 2026 leans toward allowing search and retrieval bots while giving real thought to training bots on a case-by-case basis. Check your logs before writing rules, separate training from retrieval in your robots.txt, and use server-level enforcement only where a voluntary request genuinely isn’t enough.

Compare which Plesk edition includes the SEO Toolkit and Security Core features referenced above on the Plesk Editions page.

FAQs

Does blocking AI crawlers hurt my Google ranking?
No, blocking AI-specific crawlers like GPTBot or Google-Extended does not affect Google Search rankings on its own, because Googlebot’s indexing crawl is separate from Google-Extended’s training-data signal. Blocking Googlebot itself, by contrast, would remove your site from Google Search entirely, so the two should never be confused in your robots.txt rules.

Can I block AI training but still show up in AI search answers?
Yes, for the providers that separate the two roles. Blocking GPTBot while allowing OAI-SearchBot, for example, lets you opt out of training data collection while remaining eligible for citation in that provider’s live search answers. Confirm the current bot list for each provider before finalizing rules, since these lists are updated periodically.

Is robots.txt enough to actually stop AI crawlers?
Robots.txt stops any crawler that chooses to honor it, which includes the named crawlers from major AI companies in most cases. It does not stop content that was already scraped, content published elsewhere, or a bot that simply ignores the file, which is why server-level blocking exists as a backup for content you need a firmer guarantee on.

What is llms.txt, and do I need one?
Llms.txt is a proposed file, separate from robots.txt, that points AI systems to a site’s most representative content. Adoption and actual use by major AI crawlers remain limited and inconsistent as of this writing. It costs little to create and may matter more if adoption grows, but it should not be treated as a required or high-priority control today.

Should a hosting business set one crawler policy for every client site?
No. A blanket policy across a hosting fleet ignores the fact that different client sites have different reasons to want visibility or protection. A public blog and a client’s paywalled member portal warrant different rules, so the decision is better made per subscription, informed by what that specific site’s content is and how its owner makes money from it.

Do I need to update my AI crawler rules regularly?
Yes. AI companies add new crawlers and occasionally rename existing user-agent strings, so a rule set written even a year ago may be missing bots that exist today. Reviewing crawler activity through a log analyzer every few months is a reasonable cadence for most sites.

No comment yet, add your voice below!

GET LATEST NEWS AND TIPS

The newsletter form is rendering here