The Complete Guide to robots.txt and AI Crawlers

The robots.txt file is a plain-text file at the root of your domain that tells automated crawlers which parts of your site they may access. In an AI-mediated world, it has quietly become one of the most consequential files you control, because it now governs whether AI crawlers can reach your content at all — and content an AI crawler cannot reach cannot be trained on, retrieved, or cited. A single overly broad rule can make your best documentation invisible to the answer engines your customers increasingly ask first.

This guide is for documentation managers, technical SEO practitioners, and content leaders who need to make deliberate decisions about AI crawler access rather than inheriting whatever default their platform ships. It covers what robots.txt does, which AI crawlers exist and what each one is for, how AI crawler control differs from traditional search crawler control, whether to allow or block, and the specific rules and mistakes that determine your AI visibility.

What is robots.txt, and what does it have to do with AI crawlers?

robots.txt is a plain-text file, served at yourdomain.com/robots.txt, that implements the Robots Exclusion Protocol — a decades-old convention for telling automated agents which URLs they should and should not request. It matters for AI because the same file that has guided search crawlers for thirty years is now the primary mechanism for granting or denying access to the crawlers that feed AI training corpora and live answer engines.

The file works by matching a crawler's user-agent string against a set of rules and applying Allow and Disallow directives to URL paths. When a well-behaved crawler arrives, it reads robots.txt first and honors the rules it finds. This is the key limitation to understand: robots.txt is advisory, not enforcement. It is a request that compliant crawlers respect, not an access-control barrier. It does not authenticate, encrypt, or block anything at the server level, and a crawler that chooses to ignore it faces no technical obstacle. For genuinely private content, robots.txt is the wrong tool — authentication is. For steering the reputable AI crawlers that publish their user-agent tokens and honor the protocol, it is exactly the right one.

The reason this file has taken on new weight is the shift in who is reading your content. For most of the web's history, robots.txt managed one category of visitor: the search crawler indexing pages so a human could click through. Now a growing share of crawlers are AI agents that extract answers directly, and the accessibility decisions encoded in robots.txt sit at the very first stage of whether your content ever reaches them. As the framework in the AI content supply chain describes, an accessibility failure at ingestion is the most expensive failure of all, because every bit of the work that went into producing the content is wasted before an AI system ever evaluates its quality.

Which AI crawlers should you know about?

The AI crawlers that matter fall into three functional groups: training crawlers that collect content to train future models, retrieval crawlers that fetch pages live to answer a user's current question, and general-purpose crawlers whose data feeds both. Each major AI provider operates one or more distinctly named user-agents, and knowing what each is for is the prerequisite to writing rules that reflect your actual intent rather than a blanket yes or no.

The distinction between training and retrieval access is the single most important one, because the two have opposite implications for your visibility. Blocking a training crawler affects whether future models learn about your brand from your content. Blocking a retrieval crawler affects whether an answer engine can cite you right now when a user asks a question your content answers. The mechanics of these two pathways are covered in depth in understanding AI training data and in how each AI engine retrieves content differently.

The table below summarizes the most commonly encountered AI crawler user-agents and their primary purpose. Providers add and rename tokens over time, so treat this as an orientation rather than a permanent registry, and verify the current tokens against each provider's own crawler documentation before you finalize rules.

Two entries in that table deserve special attention because they behave differently from the rest. Google-Extended and Applebot-Extended are not crawlers at all — they are control tokens. Google-Extended does not fetch any pages itself; it is a signal you place in robots.txt to tell Google whether your already-crawled content may be used for Gemini training and AI grounding. Blocking it does not affect Googlebot or your ordinary search ranking in any way. This decoupling lets you keep full search visibility while opting out of AI training, and it is a distinction many teams miss.

How is controlling AI crawlers different from controlling search crawlers?

Controlling AI crawlers differs from controlling search crawlers in what a block actually costs you. Blocking Googlebot removes you from search results — a clear, measurable, usually unacceptable loss. Blocking an AI crawler removes you from a channel whose value is harder to see on a dashboard but is growing fast, which makes the decision easier to get wrong through inattention rather than intent.

Three differences change the calculus. First, the value of AI access is often invisible in conventional analytics. A traditional crawler block shows up as lost sessions; an AI crawler block shows up as nothing, because the citations and brand impressions you forfeit never registered as traffic in the first place. Second, AI access splits into training and retrieval, so a single allow-or-block decision is really two decisions with different time horizons — training influences models months out, retrieval influences answers today. Third, the crawler population is expanding and mutating far faster than the stable set of search crawlers ever did, which means a robots.txt file that was correct last year may be silently blocking or admitting crawlers that did not exist when it was written.

There is also a newer, complementary standard worth understanding alongside robots.txt. Where robots.txt answers the binary question of whether a crawler may access a path, the emerging AI.txt standard for AI crawlers aims to express richer preferences — how content should be attributed, whether it may be used for training versus retrieval, and how often it should be refreshed. AI.txt is not yet universally adopted, so robots.txt remains the load-bearing file today, but the two are designed to coordinate rather than compete.

Should you allow or block AI crawlers?

For most organizations whose goal is visibility, the answer is to allow the retrieval and search crawlers that make you citable, and to make training access a deliberate choice rather than a default. If being named in AI answers matters to your brand — and for any content-led or documentation-led business it increasingly does — blocking the crawlers that feed those answers is choosing invisibility in the channel that is growing while traditional search contracts.

The reasoning follows directly from how citation works. As detailed in how AI answer engines choose which sources to cite, an engine can only cite content it can reach. A retrieval crawler blocked in robots.txt cannot fetch your page to answer a live question, and a training crawler blocked before a training cycle means the model never learns your brand from your own words — it fills the gap with whatever else it found, which is often a competitor or an outdated third-party description. The compounding dynamic makes early access especially valuable: models build durable associations between domains and topics, so the sources they learn to trust first tend to hold that position across cycles.

There are legitimate reasons a specific organization might block certain crawlers, and an honest guide should state them. Publishers whose business model depends on paywalled subscriptions may reasonably block training crawlers to protect the commercial value of their archive. Sites with content-licensing ambitions may withhold training access to preserve leverage in future negotiations. Organizations with strict compliance or confidentiality obligations may block AI training on principle. The framework below separates these cases by intent.

The one posture that is almost never correct is the accidental block — a blanket Disallow inherited from a template or a staging configuration that silently excludes every crawler, AI and search alike, from content you actually want discovered. Verifying that you are not in this state is a core part of any documentation AI readiness audit.

How do you write robots.txt rules for AI crawlers?

You write AI crawler rules the same way you write any robots.txt rule: a User-agent line names the crawler, and one or more Allow and Disallow lines specify the paths it may or may not request. The syntax is simple; the discipline is in being specific about which crawler each block targets so that a rule meant for one does not silently apply to all.

A block that allows every crawler full access — the right default for most public documentation — is simply an empty disallow applied to all agents:

To allow general access while blocking one specific AI training crawler, name that crawler in its own group. Rules are matched by the most specific user-agent block, so a named group overrides the wildcard for that crawler:

To allow live retrieval while opting out of training — the posture most content-led teams should consider — allow the retrieval and search agents and disallow the training agents by name:

A few rules of the syntax prevent the most common errors. Disallow: with nothing after it means "allow everything"; Disallow: / means "block the entire site" — a one-character difference with opposite meanings. Paths are case-sensitive and matched as prefixes, so Disallow: /admin blocks /admin, /administrator, and everything beneath. Include a Sitemap directive so crawlers that do have access can discover your full URL set efficiently. And remember that the Crawl-delay directive, sometimes used to rate-limit aggressive crawlers, is honored inconsistently across agents and ignored by several major ones, so it should not be relied on as a control.

What are the most common robots.txt mistakes that hurt AI visibility?

The most damaging robots.txt mistakes are the ones that block content you meant to expose — and because AI access failures are invisible in standard analytics, they can persist for months before anyone notices the missing citations. Each of the following is common, and each is fixable in minutes once identified.

• Inheriting a blanket disallow. A Disallow: / under User-agent: *, often carried over from a staging environment or a restrictive template, blocks every compliant crawler from the entire site. This is the single most expensive error, because it forecloses both search and AI visibility at once.
• Blocking retrieval when you meant to block training. Teams that decide to opt out of AI often reach for a broad block that catches the live-retrieval agents too, silently removing themselves from the answers those engines generate in real time. Separate the two intents explicitly.
• Assuming robots.txt secures private content. robots.txt is advisory. Confidential material listed as Disallow is still served to anyone — human or non-compliant crawler — who requests the URL directly, and ironically the disallow line itself advertises the path's existence. Use authentication for anything that must stay private.
• Letting the file go stale. New AI crawler user-agents appear regularly. A robots.txt written a year ago makes no decision at all about agents introduced since, defaulting them to whatever the wildcard rule says — which may not be what you would choose deliberately. Review the file on a scheduled cadence.
• Conflicting with the platform default. Many hosted documentation and CMS platforms generate their own robots.txt, and a custom file may be ignored or merged unpredictably. Confirm what is actually served at /robots.txt in production rather than what you believe you configured.

Blocking crawlers is also not the only way content becomes unreachable. A page that is technically allowed in robots.txt but renders its body only through client-side JavaScript is effectively invisible to crawlers that do not execute scripts — a separate accessibility failure covered among the crawlability checks in the guide to evaluating AI readiness in knowledge base platforms. robots.txt controls permission; server-side rendering controls whether the permitted crawler finds anything to read.

How does robots.txt fit into a broader AEO strategy?

robots.txt is the access layer of Agent Engine Optimization — the gate that determines whether every downstream signal even gets a chance to matter. Clean structure, factual density, and consistent terminology are what make content citable once a crawler reaches it, but none of that work counts if the crawler is turned away at the door. Access is necessary but not sufficient, and robots.txt is where the necessary part is decided.

The practical sequence is to get access right first, then invest in the qualities that earn citation. Confirm that the AI crawlers you want are allowed, that private content is protected by authentication rather than by an advisory line, and that your file makes a deliberate decision about training versus retrieval. Then turn to the substance: the structural and editorial standards laid out in what makes documentation AI-ready, and the broader discipline described in the complete guide to Agent Engine Optimization. For the vocabulary to coordinate this work across engineering, content, and leadership, the AEO glossary defines the terms a team needs to operate from one playbook.

The file itself is unglamorous — a handful of lines of plain text that most organizations set once and never revisit. That is precisely why it is worth revisiting now. A robots.txt written for a world of search crawlers, left untouched as AI crawlers proliferated, is making decisions about your AI visibility that no one on your team actually made. Auditing it deliberately, deciding your training and retrieval postures on purpose, and keeping it current as the crawler population evolves is one of the lowest-effort, highest-leverage moves available in an AI-first content strategy. The brands whose content AI systems reach for in the years ahead will be the ones that made sure, at the level of a single plain-text file, that the door was open.