Access logs can show which automated clients requested your pages. Review their identities and purposes before changing robots.txt.
Many websites are partially or completely invisible to AI agents — not because their content is bad, but because their robots.txt file explicitly blocks AI crawlers from accessing the site.
In this guide, we'll show you exactly which AI crawlers are visiting your site, how to check if you're blocking them, and how to fix your robots.txt so AI agents can find, understand, and do business with you.
Distinguish Crawling from Agent Actions
The web is shifting from human-first browsing to agent-first interaction. AI agents are already comparing services, booking appointments, and making purchase recommendations — all without ever opening a traditional browser.

SEO Era vs AI Agent Era
| Dimension | SEO Era | AI Agent (AX) Era |
|---|---|---|
| Audience | Search engine crawlers (Googlebot) | AI agents (GPTBot, ClaudeBot, Gemini) |
| Goal | Rank on page 1 of Google | Be usable by AI agents |
| Key File | robots.txt + sitemap.xml | robots.txt + llms.txt + MCP endpoint |
| Data Format | Meta tags, title tags | Schema.org JSON-LD structured data |
| Conversion | User finds you, fills a form | Agent finds you, books directly |
| Metric | Google Lighthouse Score | AX Score (Agent Experience) |
More bookings are starting with an AI agent. If your website blocks AI crawlers, you're not just missing out on SEO — you're becoming invisible to an entirely new channel of customer acquisition.
Crawler and Retrieval Controls to Review
These names have different purposes, and some are control tokens rather than HTTP crawlers. Verify current provider documentation; this list is not evidence that they visit your site.
12 Major AI Crawlers You Need to Know
| Crawler Name | Owner | Purpose | Importance |
|---|---|---|---|
| GPTBot | OpenAI | Training-related crawler, separate from search | Review content-use policy |
| ChatGPT-User | OpenAI | Real-time browsing when ChatGPT users ask questions | Critical |
| ClaudeBot | Anthropic | Trains Claude models, powers Claude's knowledge | Critical |
| Google-Extended | Robots.txt control for specified Gemini uses, separate from Search | Review current documentation | |
| OAI-SearchBot | OpenAI | Powers SearchGPT and ChatGPT search features | High |
| PerplexityBot | Perplexity | Powers Perplexity AI search engine | High |
| Claude-User / Claude-SearchBot | Anthropic | User-triggered retrieval and search-related crawling | Verify current provider rules |
| Gemini-Deep-Research | Deep research mode in Gemini for comprehensive analysis | Medium | |
| Applebot-Extended | Apple | Powers Apple Intelligence and Siri AI features | Medium |
| meta-externalagent | Meta | Powers Meta AI across Facebook, Instagram, WhatsApp | Medium |
| Bytespider | ByteDance | Trains TikTok's AI recommendation systems | Low |
| Amazonbot | Amazon | Powers Alexa and Amazon's AI shopping assistant | Low |
Training controls are not recommendation switches. Review OpenAI's bot documentation and Google's crawler documentation to distinguish training, Search, and user-triggered retrieval.
How to Check If Your Website Is Blocking AI Crawlers
There are three ways to check, from quickest to most thorough:
Method 1: Inspect robots.txt
Open your browser and go to yourdomain.com/robots.txt. Look for any of these patterns that block AI crawlers:
Common robots.txt Rules That Block AI Crawlers
| Rule | What It Does | Impact |
|---|---|---|
| User-agent: GPTBot / Disallow: / | Disallows GPTBot | Does not by itself block OAI-SearchBot |
| User-agent: ClaudeBot / Disallow: / | Disallows ClaudeBot | Does not describe every retrieval path |
| User-agent: Google-Extended / Disallow: / | Controls documented Gemini uses | Separate from Search inclusion |
| User-agent: * / Disallow: / | Blocks ALL crawlers including AI | Invisible to everything |
| User-agent: CCBot / Disallow: / | Blocks Common Crawl (used by many AI) | Reduced AI training data |
If you see any "Disallow: /" rules targeting AI user-agents, your site is blocking those crawlers.
Method 2: Inspect AX Audit Findings
The fastest way to get a complete picture is to use Dashform's free AX Audit tool. Enter a public URL to receive an AX Score and supporting Findings. Read the current report's checks and limits; it is not a bot-traffic log or a complete transaction test.
Use applicable Findings to plan changes, then test them on your site.
Method 3: Check Server Logs (Advanced)
If you have access to your server logs, search for AI crawler user-agent strings. Inspect relevant HTTP user agents and verify their origin using provider guidance. Google-Extended is a control token, not a separate HTTP user agent. If you see Forbidden or Too Many Requests responses, your server is actively rejecting AI crawlers.
Beyond robots.txt: 5 Other Ways Your Website Blocks AI Agents
robots.txt is just the first barrier. Many websites have additional layers that prevent AI agents from accessing content:
Hidden AI Blockers Beyond robots.txt
| Blocker | How It Works | How to Detect | Fix |
|---|---|---|---|
| CAPTCHA / Bot Walls | reCAPTCHA, hCaptcha, Cloudflare Turnstile intercept all automated requests | Visit your site in incognito, check for challenges | Allow-list AI crawler IPs or use challenge-free verification |
| JavaScript-Only Rendering | SPA frameworks (React, Angular, Vue) render content client-side that crawlers can't execute | View page source — if body is mostly empty, it's JS-rendered | Implement server-side rendering (SSR) or static site generation (SSG) |
| Cookie Consent Walls | EU cookie banners block content until accepted — agents can't click 'Accept' | Check if content is hidden behind a consent modal | Serve content first, consent banner as overlay |
| Login Walls / Paywalls | Content behind authentication is completely invisible to crawlers | Check if key pages require login to view | Offer summary content publicly, gate premium content |
| Aggressive Rate Limiting | Server returns Too Many Requests errors when crawl rate is too high | Check server logs for Too Many Requests responses to bot user-agents | Set reasonable rate limits for known bots |
How to Fix Your robots.txt for AI Crawlers (Copy-Paste Templates)
Here's exactly what to add to your robots.txt to allow AI crawlers while maintaining control:

Option 1: Explicitly Allow Selected Crawlers
Add these lines to your robots.txt file to explicitly allow all major AI crawlers:
User-agent: GPTBot / Allow: / ... User-agent: ClaudeBot / Allow: / ... User-agent: Google-Extended / Allow: / ... User-agent: PerplexityBot / Allow: / ... User-agent: OAI-SearchBot / Allow: / ... User-agent: ChatGPT-User / Allow: /
Option 2: Allow AI Crawlers But Protect Sensitive Pages
Protect private pages with authentication and authorization. robots.txt requests crawler behavior and exposes the paths it lists; it is not access control. Apply path rules to public content according to your policy.
Option 3: Allow Only Specific AI Crawlers
If you want to be selective — for example, allowing ChatGPT and Claude but not training data crawlers — you can explicitly allow certain user-agents while blocking others. Review separate search and user-triggered agents. GPTBot and ClaudeBot are training-related crawlers, so allowing them is not an example of excluding training.
Crawlers may cache policies. Monitor permitted access after the change; no indexing or recommendation timeline is guaranteed.
Beyond robots.txt: The Complete AI Readiness Checklist
Fixing your robots.txt is step one. But a truly AI-ready website needs more. Here's the complete checklist:
Complete AI Readiness Checklist
| Category | Action | Priority | Impact |
|---|---|---|---|
| Crawlability | Review each relevant crawler policy | Policy-dependent | Match access to the crawler's purpose |
| Crawlability | Test loading with the intended client | High | Observe actual failures and timeouts |
| Crawlability | No CAPTCHA or bot walls blocking AI | Critical | Blocks AI interaction entirely |
| Structured Data | Add JSON-LD Schema.org markup | Critical | Agents need structured data to understand your business |
| Structured Data | Include business type, services, hours, location | High | Enables agent-to-business matching |
| Structured Data | Validate OpenGraph tags (title, description, image) | Medium | Improves how agents present your business |
| Content | Semantic HTML (proper headings, landmarks) | High | Agents parse structure, not just text |
| Content | Alt text on your images | Medium | AI agents read alt text for context |
| Agent Interaction | Create an llms.txt file | High | The new standard for AI-readable site summaries |
| Agent Interaction | Set up MCP endpoint (.well-known/mcp.json) | Medium | Enables agents to take actions on your site |
| Agent Interaction | Clear CTA labels (not vague 'Click here') | Medium | Agents need descriptive action labels |
| Discoverability | Schema.org sameAs links to social profiles | Medium | Cross-references verify your identity |
| Security | HTTPS enabled with valid SSL certificate | Critical | Agents won't trust insecure sites |
| Security | Privacy policy and terms of service linked | Low | Trust signals for AI evaluation |
Check Your Score: Free AX Audit Tool
Instead of manually checking each of these items, run a free AX Audit on your website. It provides an AX Score, Readiness Layers, and Findings. Read the current report's checks and limitations.
Findings describe observed conditions and may include fixes. The audit does not change the website or guarantee a working transaction.
AX Score Grading Scale
| Grade | What It Means |
|---|---|
| Excellent | Strong performance on assessed conditions; test actual agent actions separately |
| Good | Review remaining Findings and test the intended client |
| Needs Work | Investigate the reported technical gaps |
| Poor | Prioritize reported gaps; the grade does not measure recommendation frequency |
Frequently Asked Questions
What is robots.txt and why does it affect AI agents?
robots.txt is a text file at the root of your website (yourdomain.com/robots.txt) that tells web crawlers which pages they can and cannot access. Traditionally it was used to guide search engine bots like Googlebot. Now AI companies use their own crawlers — GPTBot, ClaudeBot, PerplexityBot — that also respect robots.txt rules. A rule applies to the named crawler; blocking a training crawler does not block every search or user-triggered retrieval path.
Will allowing AI crawlers hurt my SEO?
Review crawler-specific purposes and site capacity before changing access. Google-Extended is separate from Google Search crawling; allowing training-related access does not guarantee a ranking or recommendation benefit.
How do I know which AI crawlers are visiting my site?
Inspect access logs and verify crawler identity. Google-Extended is not a separate HTTP user agent. AX Audit checks declared policies, not which bots actually visited.
Is it safe to allow AI crawlers? What about content scraping?
Allowing AI crawlers lets them read your publicly available content — the same content any human visitor can see. It does not give them access to private data, admin areas, or anything behind authentication. You can use selective Allow/Disallow rules to open public pages while keeping sensitive directories blocked. For most businesses, being visible to ChatGPT's users outweighs concerns about content being used in AI training.
How long after fixing robots.txt will I appear in AI tools?
Crawlers may cache robots.txt and revisit on their own schedules. A policy change does not guarantee indexing, citations, or an appearance timeline. Verify access and observe the intended retrieval behavior separately.
What is an AX Score?
AX Score (Agent Experience Score) is a single score for how well your website is prepared for AI agents — similar to how Google Lighthouse measures performance and accessibility. It summarizes the current audit's weighted technical checks, not recommendation frequency, traffic, or revenue. You can check your AX Score for free at getaiform.com/axaudit.
Do I need technical skills to fix these issues?
Basic robots.txt changes are straightforward text edits that anyone can do. More advanced fixes like adding JSON-LD structured data or creating an llms.txt file require some technical knowledge, but the AX Audit tool provides copy-paste code snippets for every issue it finds. For businesses that want a fully agent-ready web presence without technical work, Dashform provides MCP-discoverable forms and AI-native funnels that are agent-ready by default.
Key Takeaways
Many websites are partially or completely invisible to AI agents because their robots.txt blocks AI crawlers — this is fixable in minutes
Crawler purposes differ. Verify training, search, and user-triggered controls separately.
robots.txt is just the first layer — CAPTCHAs, JavaScript-only rendering, cookie consent walls, and aggressive rate limiting can all silently block AI agents
The complete AI readiness checklist covers 6 dimensions: Crawlability, Structured Data, Content Quality, Agent Interaction, Discoverability, and Security
A policy change does not guarantee recommendations. Monitor observed access and behavior.







