Should you block GPTBot? What changes, and what stays open?

What result are you trying to get?

A small publisher might have one concern about training and another about distribution. Keeping its latest reporting available to search can serve readers even when the publisher declines automatic training collection. There is no need to give those choices the same answer.

The table is the decision I would write before opening a hosting dashboard. It prevents a crawler preference from being mistaken for a privacy control or a promise about where the publication will appear.

What does a GPTBot disallow change?

GPTBot is the automatic crawler whose documented purpose includes gathering material that may be used for OpenAI foundation-model training. A disallow communicates your choice for that use. OpenAI’s search agent has a different token, and ChatGPT-User handles certain user-initiated actions; OpenAI says robots rules may not apply to the latter. OpenAI’s agent definitions.

This makes the wording of a change request matter. “Decline GPTBot collection of public essays” is implementable. “Make the site disappear from AI” leaves unanswered which service, which workflow and which copy of the material the owner means.

If the concern is sensitive information, move the decision to the server. A URL listed under Disallow remains a publicly disclosed path, and an HTTP client can request it without following the file. RFC 9309 defines the protocol as crawler rules rather than authorization. The practical boundary is covered in why robots.txt cannot protect a private page.

How do you keep search open while blocking GPTBot?

This complete example starts with a public site that excludes its internal search and previews from ordinary crawling. It repeats those exclusions in the named search group so that adding the group does not reopen them. Replace the example hostname and reconcile every existing rule before using it.

The repeated paths are deliberate. Under the protocol, a matching named group supplies the crawler’s rules; the wildcard group is the fallback. Two groups for the same product token combine, so appending another GPTBot group is not a reliable way to overwrite an earlier policy. RFC 9309 grouping rules.

User-agent: *
Disallow: /search/
Disallow: /preview/

User-agent: OAI-SearchBot
Allow: /
Disallow: /search/
Disallow: /preview/

User-agent: GPTBot
Disallow: /

Sitemap: https://example.com/sitemap.xml

What would count as a successful change?

I would accept the policy change when the public host serves the intended file and a parser gives the intended answer for both an article and an excluded path. A repository diff is only the first half of that check: a CMS, cache or CDN can serve another version.

Access deserves its own check. Preserve a normal GET response for a public article, including the body. If a real OAI-SearchBot request later reaches the edge, join its identity evidence to the delivered response using the server-log verification method. A curl request that copies the bot’s name comes from your network and tests only that synthetic request.

Write down the observation window. No attributed GPTBot request after a change means no attributed request in that window; the bot may simply have had no reason to return. Conversely, a denied request can be evidence that the WAF acted, while telling you little about whether the crawler had already read the new policy.

• Save the old and new public robots.txt bodies with the deployment time.
• Check GPTBot against a public article and OAI-SearchBot against both allowed and excluded URLs.
• Inspect a returned article body for the expected text, login screens or challenges.
• Retain any later attributed request with its timestamp, response and policy version.

When would I leave GPTBot allowed?

An owner may actively want its public material used in model development, or have an agreement that makes collection worthwhile. Leaving the crawler allowed can be a deliberate choice. The useful question is whether that use fits the publication’s rights and business model, not whether every other site has clicked the same switch.

I would judge the decision using the owner’s goals and separately measured costs or referrals. The checked documentation gives no basis for promising a ranking reward for permitting GPTBot, or a ranking penalty for blocking it. For the broader map of providers and workflows, use the AI crawler policy guide. My default for the publisher described here remains simple: keep the public search route working and make the training choice explicit.