Can ChatGPT Read Your Website? Check This First

ChatGPT Search may need to visit your website before it can quote or summarize what is on the page. OpenAI calls the crawler used for that job OAI-SearchBot. If your site blocks it, you have closed one route into ChatGPT's search answers.

Opening that route does not earn a mention, citation or recommendation. It only removes an access problem. Before anyone sells you a new robots.txt line, check whether the crawler is blocked in the first place.

The setting controls access, not preference

OpenAI says websites that opt out of OAI-SearchBot will not appear inside ChatGPT search answers, although a page title and link may still show up as a navigational result. Its publisher guidance tells site owners who want their content included in summaries and snippets to avoid blocking that crawler.

This is an eligibility check. A readable service page can still be ignored when it does not answer the question, lacks current local detail, or is weaker than the sources retrieved for that particular search. OpenAI does not offer a top-placement guarantee for adding an allow rule.

Search and training are separate choices

The names are easy to mix up. OAI-SearchBot is the control for appearing in ChatGPT's search results. GPTBot is the control for content that may be used to train OpenAI's foundation models. OpenAI documents the two settings as independent.

A business can therefore allow the search crawler and disallow the training crawler. That is a policy choice, not an SEO trick. ChatGPT-User is different again: it may visit a page because a person asked ChatGPT to do something, and OpenAI says it is not used to decide whether a page may appear in Search.

OAI-SearchBot

Controls automatic search crawling. Allow it if you want public pages eligible to be surfaced with content in ChatGPT Search. A robots.txt change may take about 24 hours to be reflected in OpenAI's systems.

GPTBot

Controls potential training use. Disallowing it signals that crawled content should not be used to train OpenAI's generative foundation models. This can be set independently from search access.

ChatGPT-User

Handles some user-requested visits. It is not an automatic search crawler, and OpenAI says robots.txt rules may not apply to these user-initiated actions.

What 11 local business robots files showed

On August 22, 2026, I checked the public robots.txt files of 11 Moncton-area businesses found through live searches: four roofing companies, three dental practices, and four restaurants or food-market sites. This was a small convenience sample, not a measure of all local businesses or website platforms.

None of the 11 files named OAI-SearchBot. None used a blanket rule that blocked every crawler from the whole site. Under normal robots.txt matching, an unnamed crawler follows the applicable general rules, so none of these files showed a full-site OAI-SearchBot block. One site explicitly blocked GPTBot from the whole site without placing the same full-site block on OAI-SearchBot.

Check the file before you edit it

Open your website address and add /robots.txt to the end. Search the file for OAI-SearchBot, then look for a general User-agent: * section. You are looking for a Disallow: / rule that applies to the whole site or for narrower rules that cover pages customers need to see.

Do not paste a block of crawler code from an article if the current file already allows access. Some website platforms generate robots.txt automatically, and a manual change can be overwritten or create a conflict. Send the file to whoever manages the website when the rules are unclear.

Check the homepage, key service pages and location pages rather than assuming one rule covers every useful URL
Keep account, cart, search-result and private areas out of scope when they do not belong in public search
Decide separately whether GPTBot should be allowed for potential training use
Save a dated copy of the old file before changing a platform-managed setting

A clean robots file can still hide a locked door

A recent TechSEO discussion supplied a useful field warning: people reported sites whose robots.txt looked open while a content-delivery network or firewall returned a 403 error to AI crawlers. That thread is anecdotal, so it is not proof of how often this happens.

The failure mode itself is supported by OpenAI's crawler guidance. OpenAI warns that firewalls, bot mitigation, JavaScript challenges and CAPTCHAs can deny automated access even when robots.txt permits the visit. For most owners, the practical evidence lives in server or security logs: successful requests, 403 errors, CAPTCHA challenges and rate limits tied to a verified crawler.

Do not treat the user-agent name as proof. Those strings can be copied. OpenAI publishes IP ranges for OAI-SearchBot so a technical partner can verify that a request really came from its crawler.

Read the result as a sequence

Crawler access sits at the start of a longer chain. The page must be reachable, relevant to the question, selected as a useful source, and then used in a way the customer can see. A recommendation adds another judgment: whether the business appears to fit the service, place and constraint in the question.

This is why an open crawler and a missing recommendation can both be true. Once access is confirmed, move to the page itself and the public evidence around the business. Repeating the same customer questions after a change will tell you more than repeatedly editing robots.txt.

Reachable: the crawler is permitted and the server lets it through
Relevant: the page clearly answers the customer's service and location question
Retrieved: the product selects the page for that particular search
Cited: a visible source link connects the page to the answer
Recommended: the business is presented as a suitable choice