Check access logs and representative client behavior to establish whether and how automated clients visit your site. This article does not assume a visit count or a population-wide visibility rate.
Access restrictions, rendering requirements, and missing structured information can affect particular clients. AI-powered forms provide one possible interaction surface, but technical readiness does not establish lost or gained leads.
This guide will show you exactly how AI agents see your website today, what is blocking them, and the step-by-step fixes to become fully visible.

Evaluate Agent Access Alongside Search
For 20 years, search engine optimization (SEO) was the gold standard for online visibility. Businesses optimized for Google's crawlers, fought for page-one rankings, and measured success in organic traffic. But in 2026, the game has fundamentally changed.
An assistant may use search, browsing, or connected tools for a service request. Checking insurance and confirming a booking require appropriate data sources, supported tools, and authorization; they are not automatic consequences of a prompt.
| Dimension | Traditional SEO | AI Agent Visibility |
|---|---|---|
| Primary Audience | Googlebot, Bingbot | GPTBot, ClaudeBot, Gemini, PerplexityBot |
| Goal | Rank on page 1 of search results | Be discoverable and usable by AI agents |
| Key Files | robots.txt + sitemap.xml | llms.txt + .well-known/mcp.json |
| Data Format | HTML meta tags, title tags | Schema.org JSON-LD, structured data |
| Conversion Path | User clicks link, fills form | Agent qualifies lead, books directly |
| Success Metric | Google Lighthouse score | AX Score (Agent Experience) |
| Commercial evidence | Measure search-sourced outcomes | Measure agent-sourced outcomes separately |
Treat agent access as a task to test, and assess its commercial contribution from your own records.
The 5 Barriers Making Your Website Invisible to AI

1. robots.txt Is Blocking AI Crawlers
Start by inspecting access rules. Your robots.txt file -- originally designed to guide Google's crawler -- may be actively blocking every major AI agent. Many default CMS configurations and security plugins add blanket disallow rules that block GPTBot, ClaudeBot, and other AI crawlers.
| AI Crawler | Company | robots.txt User-Agent | Impact If Blocked |
|---|---|---|---|
| GPTBot | OpenAI | GPTBot | Restricts this crawler; does not establish all ChatGPT access |
| ClaudeBot | Anthropic | ClaudeBot | Restricts this crawler; other access paths can differ |
| Google-Extended | Google-Extended | Excluded from Gemini AI training | |
| PerplexityBot | Perplexity | PerplexityBot | Invisible to Perplexity AI search |
| Bytespider | ByteDance | Bytespider | Review the crawler operator's current policy |
| Amazonbot | Amazon | Amazonbot | Restricts this crawler; does not establish all assistant access |
Quick check: Visit yourwebsite.com/robots.txt right now. If you see "Disallow: /" under any AI user-agent, you are blocking that agent completely. Read our complete robots.txt guide for step-by-step fix instructions.
2. JavaScript-Heavy Sites That AI Cannot Render
Single-page applications (SPAs) built with React, Angular, or Vue often serve an empty HTML shell that requires JavaScript execution to render content. Most AI crawlers do not execute JavaScript -- they see a blank page where your business information should be.
If your website relies on client-side rendering without server-side rendering (SSR) or static site generation (SSG), AI agents see nothing useful. This is especially common with modern JavaScript frameworks.
3. CAPTCHA and Bot Protection Walls
Security tools like reCAPTCHA, hCaptcha, Cloudflare Turnstile, and DataDome are designed to block bots -- and that is exactly what AI crawlers are. These tools cannot distinguish between a malicious scraper and an AI agent trying to understand your business to recommend you to potential customers.
4. Missing Structured Data (Schema.org)
Even if AI agents can access your site, they struggle to extract meaningful business information without structured data. Schema.org JSON-LD markup tells AI agents exactly what your business does, your services, pricing, location, and hours. Without it, agents are guessing from unstructured HTML.
5. No Machine-Readable Business Summary
An llms.txt file is an optional documentation proposal that links to useful site content. It is not required for every assistant and does not guarantee discovery.
How to Check Your Website's AI Visibility Right Now

Use Dashform's AX Audit to inspect technical readiness. The report groups weighted checks into Readiness Layers; it does not measure actual AI visibility or traffic.
Readiness Layers
| Readiness Layer | Included dimensions |
|---|---|
| Discovery | Discoverability and freshness |
| Identity | Structured data, trust signals, and authoritativeness |
| Access | Crawlability, reachability, performance, and security/trust |
| Agent integration | Agent interaction and actionability |
| User experience | Content quality and usability |
See how the AX Audit works for current product-defined weights and limitations.
Understanding Your AX Score
| Grade | What It Means |
|---|---|
| Excellent (Agent-Ready) | Strong performance on the audit's checks; no transaction guarantee |
| Good (Mostly Ready) | Review remaining Findings and test intended clients |
| Needs Work | Significant gaps that limit AI agent engagement |
| Poor (Not Ready) | Substantial gaps under the rubric; inspect their relevance |
Step-by-Step: Making Your Website Visible to AI Agents
Step 1: Review Your robots.txt
Add explicit allow rules for the major AI crawlers. At minimum, add these lines to your robots.txt:
User-agent: GPTBot / Allow: / -- This allows OpenAI's crawler to index your site. Repeat for ClaudeBot, Google-Extended, PerplexityBot, and other AI crawlers listed above. Keep your existing Google and Bing rules unchanged.
Step 2: Add Accurate Schema.org Structured Data
Add JSON-LD markup to your homepage and key service pages. At minimum, include LocalBusiness or Organization schema with your business name, description, address, phone, hours, and services. This gives AI agents structured access to your core business information.
Step 3: Consider an llms.txt File
Create a Markdown file at yourwebsite.com/llms.txt that summarizes your business for AI agents. Include your business name, description, services, contact information, and key links. See our complete llms.txt setup guide for the exact format and examples.
Step 4: Configure a Supported Agent Workflow
The highest level of AI visibility is not just being found -- it is being actionable. This means AI agents can qualify leads and book appointments directly, without requiring a human to fill out a form. Dashform's Agent Funnel turns your lead funnel into an MCP endpoint that any AI agent can discover and interact with -- one toggle, zero code.
Step 5: Re-Audit and Monitor
After making changes, run AX Audit again to verify your improvements. AI agent technology evolves rapidly, so re-audit monthly to catch new issues. The score effect depends on which Findings changed; verify completed actions separately.

Illustrative Checks by Industry
These examples suggest what to inspect. They are not benchmarks or findings from a measured scan population:
| Industry | Common Issue | Quick Win |
|---|---|---|
| Med Spas & Wellness | robots.txt blocking all AI crawlers | Allow GPTBot and ClaudeBot |
| Dental & Healthcare | No Schema.org structured data | Add LocalBusiness JSON-LD |
| Real Estate | JavaScript-heavy listing pages | Add SSR or pre-rendering |
| Restaurants | Missing business hours in structured data | Add Restaurant schema |
| Legal Services | CAPTCHA walls blocking all bots | Allow AI user-agents |
| Fitness & Gyms | No llms.txt or machine-readable services | Create llms.txt file |
| E-commerce | Product data not in JSON-LD format | Add Product schema markup |
| SaaS & Technology | No MCP or API discoverability | Add .well-known/mcp.json |
Choose fixes from your own Findings and test the intended client; their effort and effect vary.
Frequently Asked Questions
How do I know if AI agents are visiting my website?
Check your server access logs for user-agent strings containing GPTBot, ClaudeBot, Google-Extended, PerplexityBot, or other AI crawler names. Most analytics tools do not track these visits separately, so server logs are the most reliable method. You can also use Dashform's AX Audit to check which crawlers can access your site.
Will making my site visible to AI agents hurt my SEO?
No. AI visibility improvements are complementary to SEO, not competitive. Adding Schema.org structured data, improving semantic HTML, and creating llms.txt all strengthen your existing SEO while adding AI agent compatibility. Google's own algorithms increasingly favor well-structured, machine-readable content.
Do I need to change my website design for AI agents?
No visual changes are needed. AI visibility improvements are entirely technical -- robots.txt rules, structured data in your HTML head, a text file at your domain root, and optional API endpoints. Your human visitors will not notice any difference.
How long does it take to become AI-visible?
Time and score changes depend on the site and the Findings. Review access, markup, rendering, and supported actions separately, then re-run the audit and test the intended client.
What is the difference between AI visibility and AI optimization?
AI visibility ensures agents can find and understand your business. AI optimization (or AEO -- Agent Experience Optimization) goes further by enabling agents to transact on your behalf -- qualifying leads, booking appointments, and processing orders. Visibility is the foundation; optimization is the competitive advantage.
Is Dashform's AX Audit really free?
AX Audit offers a public readiness scan without signup. Supported targets and service limits apply; it does not measure actual assistant citations or promise a scan completion time.
Check Your AI Visibility Score Now
Identify the actual access or interaction problem before choosing a fix. Run your free AX Audit now to see exactly where you stand and get actionable fix-it instructions for every issue found. If the list is longer than your week allows, Fecto (same team, same scan) works toward more visits from AI assistants and charges only when the number moves.
Ready to go beyond visibility? Dashform's Agent Funnel turns your lead funnel into an AI-agent-discoverable service with one click. Test access with a compatible client, configure hot/warm/cold criteria, and set up any external booking service separately.







