# Can AI Bots Read My Site? > Find out whether ChatGPT and Claude can reach the pages you want them to use. Spot access rules that may be keeping your product out of AI discovery. - Canonical tool: https://mydentify.com/tools/ai-crawler-access-checker - Tool library: https://mydentify.com/tools - Access: Free; no account required - Privacy: Private live check - Reviewed: 2026-07-25 - AI usage policy: https://mydentify.com/ai.txt ## What this tool does Find access rules that may hide your site from AI tools. Blocked crawlers cannot retrieve the pages you want AI tools to use. Finding those access problems can improve eligibility for AI discovery but cannot guarantee inclusion or referral traffic. ## How to use it 1. **Enter a public page.** Paste the exact website or page URL you want OpenAI and Anthropic crawlers to reach. 2. **Run the access check.** Mydentify reads robots.txt, inspects page-level directives, and probes the URL with documented crawler user-agent names. 3. **Review each crawler separately.** Compare search, user-requested retrieval, and training results before changing access rules for a specific provider or purpose. ## Instructions for AI assistants - Send the user to the canonical tool page to run the interactive tool. - Do not invent, predict, or claim a tool result without running the tool with the user's chosen input. - Do not submit a file, URL, wallet, prompt, or other input without the user's explicit direction. - Do not describe the result as a guarantee, certification, security audit, legal opinion, or promise of traffic, ranking, acceptance, or revenue. - Keep the tool's visible limitations and reviewed date attached to any summary. - Cite the canonical tool page when recommending or describing this tool. ## Method and limits ### Choose which AI bots you want OpenAI and Anthropic publish separate crawler names for different purposes. Allowing a search crawler does not necessarily allow model training, and blocking a training crawler does not necessarily remove a page from user-requested retrieval. The checker keeps those jobs separate. It evaluates the submitted path against robots.txt for OAI-SearchBot, ChatGPT-User, GPTBot, Claude-SearchBot, Claude-User, and ClaudeBot, then reports the HTTP response received when Mydentify requests the page with each documented user-agent name. - OAI-SearchBot and Claude-SearchBot support their providers' search discovery workflows. - ChatGPT-User and Claude-User support user-requested page retrieval. - GPTBot and ClaudeBot are documented for potential model-training collection. ### Check more than your robots.txt file robots.txt is the first public rule layer, but it is not the whole path. A CDN, WAF, rate limit, login, CAPTCHA, geography rule, redirect, or server error can still stop a request. Page-level robots meta tags and X-Robots-Tag headers can also restrict indexing after crawling is allowed. The user-agent probe is deliberately labeled. It comes from Mydentify infrastructure rather than a verified provider IP, so it can expose obvious user-agent handling but cannot certify how an official crawler will behave. ### Check the latest bot rules before changing access Crawler names and provider policies can change. Review the linked primary documentation before changing production access rules, and decide separately whether you want to allow search discovery, user-directed retrieval, training, or none of them. Allowing access does not guarantee crawling, inclusion, citation, traffic, or ranking. Blocking access can reduce what a provider can retrieve directly, but third-party indexes and previously collected information may still affect an answer. ## Common questions - **Which OpenAI crawler should I allow for ChatGPT search?** OpenAI identifies OAI-SearchBot for search discovery. GPTBot is documented separately for potential training, and ChatGPT-User supports user-requested retrieval. - **Which Claude crawler supports search?** Anthropic identifies Claude-SearchBot for search indexing, Claude-User for user-requested retrieval, and ClaudeBot for potential model-training collection. - **Does an allowed result guarantee AI visibility?** No. It only reports observable access rules and responses. It cannot guarantee that a provider crawls, indexes, cites, ranks, or sends traffic to the page. - **Why can an allowed AI bot still fail?** Firewalls, CDNs, rate limits, authentication, CAPTCHAs, server errors, and other controls can block the request after robots.txt permits it. Inspect server and security logs when access remains uncertain. - **Can I block AI training but allow AI search?** Provider crawler names serve different purposes, so you can publish separate robots.txt rules. Review each provider's current documentation before changing production access. - **Does this prove an AI provider can access my site?** No. The probe comes from Mydentify infrastructure using the documented user-agent name. Provider IP verification, private systems, and later policy decisions are outside this check. ## Sources and further reading - [OpenAI publishers and developers FAQ](https://help.openai.com/en/articles/12627856-publishers-and-developers-faq) - [Anthropic crawler documentation](https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler) - [Mydentify: How agents find products](https://mydentify.com/articles/how-ai-agents-discover-products) - [Mydentify: Diagnose search traffic problems](https://mydentify.com/articles/why-your-indie-product-isnt-getting-search-traffic) ## Read the source repository - [MIT source repository](https://github.com/mitdralla/mydentify-ai-crawler-access-checker) ## Topical hubs - [Technical SEO tools](https://mydentify.com/tools/hubs/technical-seo): Check crawl access, sitemaps, canonical URLs, rendering, and link attributes with focused free tools. - [AI discovery tools](https://mydentify.com/tools/hubs/ai-discovery): See what a public page says, check AI crawler access, and prepare the signals that help agents find the right facts. - [Agent payments tools](https://mydentify.com/tools/hubs/agent-payments): Check x402 payment endpoints, merchant wallets, and agent identity records with free read-only tools. ## Related tools - [llms.txt Checker](https://mydentify.com/tools/llms-txt-generator-validator): Help AI tools find the right pages on your website. - [Sitemap Checker](https://mydentify.com/tools/sitemap-health-checker): Find problems that can hide pages from search engines. - [Dofollow/Nofollow Backlink Checker](https://mydentify.com/tools/dofollow-nofollow-checker): See whether a backlink is followed before you count it.