Should You Make Your Help Center Public for AI Assistants

The benefits and risks of making help center documentation publicly indexable for LLMs sit on different pages of the same help center. How-to articles gain from being quoted by an assistant, while a partner price list or an escalation runbook is damaged the moment an anonymous visitor can open it. So the working decision is made per content type, with three controls that do three different jobs: a robots.txt rule for each crawler, a noindex tag, and a login.

What does public support documentation gain in AI answers?

Making support documentation public benefits AI search and LLM visibility in one direct way: the pages become eligible to be retrieved, quoted and linked. Each operator states that condition in its own documentation. The OpenAI Help Center says "Any public website can appear in ChatGPT search" and tells publishers who want their content in ChatGPT summaries and snippets to make sure they are not blocking OAI-SearchBot. Google Search Central sets the bar for a supporting link in AI Overviews or AI Mode at a page that is "indexed and eligible to be shown in Google Search with a snippet", and adds on the same page that "Indexing and serving isn't guaranteed." Anthropic warns that disabling Claude-SearchBot "may reduce your site's visibility and accuracy in user search results."

The commercial case follows from who is asking. A customer who asks an assistant how to do something and gets your article back may never open a ticket, which bears on support cost. An evaluator asks whether the product handles a specific integration, limit or compliance need, and help articles are where those details get written down. A fully gated help center leaves both groups with whatever third parties have published about you, the starting problem in getting cited by ChatGPT and Perplexity and the reason support content belongs in any count of AI share of voice.

What does a public knowledge base expose?

The risks of making a knowledge base public start with competitors scraping it: anything a visitor can read without signing in, a competitor's analyst or script can read as well. A robots.txt rule does nothing about that. RFC 9309, the IETF's Standards Track specification for the Robots Exclusion Protocol (September 2022, published by the RFC Editor), says of its own rules: "These rules are not a form of access authorization." It carries a second warning that matters for help centers: "Listing paths in the robots.txt file exposes them publicly and thus makes the paths discoverable." A Disallow line for an internal runbook folder works as a signpost to that folder.

In our view, most of what a competitor learns from a typical help center is what a trial account or a sales call would also reveal: feature limits, the integration list, the workarounds that hint at gaps. The costly material is whatever you would withhold from a prospect until a contract is signed, and its protection should never rest on crawler etiquette.

Two quieter exposures deserve attention. Unreviewed public articles can be quoted as current, so a retired plan limit may reach a buyer as fact; our piece on the entity consistency audit goes further into conflicting statements. Crawler rules also do not bind every fetch. OpenAI says of ChatGPT-User, the agent that may visit a page when a user asks ChatGPT a question: "Because these actions are initiated by a user, robots.txt rules may not apply."

Competitors and AI training are separate questions

Making a help center public bundles benefits and risks that have different owners: competitors are an access question, while AI training is a crawler-policy question that the large operators let you answer apart from search.

One detail behind that table changes decisions. Google describes Google-Extended as managing the use of crawled content for training future Gemini models and also for grounding in Gemini Apps and in Grounding with Google Search on Vertex AI, so disallowing it gives up more than training. AI Overviews and AI Mode sit with Googlebot, because Google Search Central treats them as part of Search.

Whether to allow training crawlers is a policy call, and we lean toward allowing them on how-to and API content: when an assistant answers without a live search, it relies on what its model learned in training, and we would rather that include the official documentation. Record the decision either way. OpenAI says it can take about 24 hours after a robots.txt update for its search systems to adjust.

Robots.txt, noindex and login each do one job

Robots.txt asks a named crawler to stay out. A noindex tag keeps a fetched page out of results in search engines that support it, and it works only on a page the crawler can reach. Google Search Central is blunt about stacking the two: "If the page is blocked by a robots.txt file or the crawler can't access the page, the crawler will never see the noindex rule, and the page can still appear in search results, for example if other pages link to it." OpenAI's publisher FAQ gives a matching instruction for its own crawler: where it learns a disallowed URL from a third-party search provider or from crawling other pages and has signals that the page is relevant to a query, it "may surface just the link and page title in ChatGPT Atlas", and the remedy it names is a noindex meta tag on a page its crawler is allowed to read.

Login is the one control of the three that holds against a person, a scraper and a user-triggered fetch. On a hosted help center these controls live in the platform's settings. Zendesk, for one, says search engines by default do not index a help center that requires sign-in, that specific articles can be restricted to signed-in users with user segments, and that a public help center needs a noindex meta tag in its theme code to stay out of results. It also warns that after the tag is added, "the index removal process can take months." Plan matters as well: Zendesk notes that on Suite Team an article can be visible to everyone or to signed-in users only, with staff-only and custom-segment restrictions unavailable.

Which setting fits each content type?

One test sorts most articles: would a support agent say this to an anonymous caller? If so, publish it. If a sales engineer would want a signed agreement first, gate it. The rows below are our recommended defaults, built on the documented behavior above.

Retired-version docs stay reachable by link for customers on old versions, and their noindex tag works only while search crawlers can fetch the page. Published list prices belong on the public pricing page, while the rules behind a quote belong behind login.

A worked example on a hypothetical help center

Take a hypothetical software company whose help center lives at help.example.com, on a platform that lets the team edit robots.txt, in four URL groups: /articles/, /api/, /legacy/ and /partners/. The partner section moves behind login. The team decides that training crawlers may read the API reference and nothing else. Because Anthropic asks site owners to set an opt-out "for every subdomain that you wish to opt out from", the file goes on the help subdomain itself:

User-agent: OAI-SearchBot
User-agent: Claude-SearchBot
User-agent: Claude-User
User-agent: Googlebot
Allow: /

User-agent: GPTBot
User-agent: ClaudeBot
Allow: /api/
Disallow: /

RFC 9309 says the most specific matching rule must be used, so the API folder stays open to the training group while the rest is closed to it. Google-Extended is left out of that group because this team wants its articles available for grounding in Gemini Apps, and it accepts that the same token also governs training of Gemini models. Pages under /legacy/ carry a noindex tag and remain crawlable. The partner folder is absent from the file on purpose, since it sits behind login and naming it would advertise it.

Approvals, failure modes and measurement

The work needs three inputs: an article inventory with type, owner, audience and last-reviewed date; server or CDN logs that record user agents; and the support and evaluation questions you want assistants to answer from your pages. Support leadership signs off the accuracy of everything public, security and legal approve each move between public and login, and marketing owns the crawler policy and records the training decision in writing.

The failures we would check for first:

• A Disallow and a noindex on the same URL, where the Disallow keeps the crawler from ever reading the noindex.
• A CDN or firewall blocking a crawler that robots.txt allows. Google Search Central lists "Ensuring that crawling is allowed in robots.txt, and by any CDN or hosting infrastructure" among its SEO fundamentals.
• Blocking Anthropic's bots by IP address, which Anthropic says "may not work correctly or persistently guarantee an opt-out".
• Public articles with no review date, which is how a stale limit ends up quoted.

For measurement, count crawler hits by token on help center paths, and segment sessions that land on help URLs with utm_source=chatgpt.com, the parameter OpenAI says ChatGPT adds to referral URLs. Google counts AI Overviews and AI Mode traffic inside the Search Console Performance report under the "Web" search type, so read help center URLs there as a blended figure. Run a fixed set of support and evaluation prompts on a schedule and log whether your help URLs are cited, with the caveats in our piece on prompt-based visibility tracking. Ticket volume on the topics you opened up is worth watching for cost, though product changes and seasonality move it too.

The AI Visibility Checker is a further resource for this review, and AI search optimization is the service where we do this work with clients.

Where we come down

Publish what a support agent would say to an anonymous caller, and let the search crawlers in. Put anything that needs a contract behind login and keep its path out of robots.txt. Treat training crawlers as a written policy choice, since OpenAI, Anthropic and Google each document a way to make it without leaving search. A single blanket setting across the whole help center, in either direction, is the choice we would argue against.

Sources

• OpenAI: Overview of OpenAI Crawlers
• OpenAI Help Center: Publishers and Developers - FAQ
• Anthropic: Does Anthropic crawl data from the web, and how can site owners block the crawler? (April 7, 2026)
• Google: Google's common crawlers (last updated July 14, 2026)
• Google Search Central: AI Features and Your Website (last updated December 10, 2025)
• Google Search Central: Block Search Indexing with noindex (last updated December 10, 2025)
• RFC Editor: RFC 9309, Robots Exclusion Protocol (IETF, September 2022)
• Zendesk: Can I restrict Google or other search engines from indexing my help center?
• Zendesk: Setting view permissions on articles with user segments