AI crawler checker: can AI bots access your website?
- robots.txt review
- GEO score included
- $2 one time
- No signup needed
AI assistants can only cite what their crawlers can read. Ranklyst reads your robots.txt file, flags rules that block AI bots, and includes the result in a full audit with a ranked fix list. It takes minutes and costs $2.
- See whether your robots.txt blocks the bots AI assistants use
- Spot catch-all rules left over from a staging site
- Get a prioritized fix list with effort and impact
$2 one-time payment. No account needed. Your report is ready in a few minutes and sent to your email.
What is an AI crawler checker?
An AI crawler checker shows whether the bots that AI companies use can reach your website. It reads your robots.txt rules, looks for blocks that stop AI crawlers, and helps you fix them. An AI crawler is an automated bot that visits web pages to train models, build search indexes for AI assistants, or fetch a page when a person asks an assistant to open it.
Access is the first gate for AI visibility. If a crawler cannot read your pages, an assistant has little to cite, however good your content is. The check is part of every Ranklyst audit and feeds the GEO score in our AI visibility checker.
Reviewed by Fahad Ahmad. Last updated . Crawler names and roles change, so confirm them in each company’s documentation.
Two things people mean by “AI crawler”
The phrase covers two different goals. Knowing which one you have saves you from searching for the wrong tool.
Training, search and user-triggered: three kinds of AI crawler bots
Most advice treats every AI bot the same. They do different jobs, and you can make a separate choice for each.
Common AI crawler bots and what each one does
| Company | Crawler name | Type | What blocking it does |
|---|---|---|---|
| OpenAI | GPTBot | Training | Opts pages out of model training |
| OpenAI | OAI-SearchBot | Search | Can keep pages out of ChatGPT search results |
| OpenAI | ChatGPT-User | User-triggered | Affects pages opened at a person’s request |
| Anthropic | ClaudeBot | Training | Opts pages out of model training |
| Anthropic | Claude-SearchBot | Search | Can keep pages out of Claude search results |
| Anthropic | Claude-User | User-triggered | Affects pages opened at a person’s request |
| Perplexity | PerplexityBot | Search | Can keep pages out of Perplexity answers |
| Perplexity | Perplexity-User | User-triggered | Affects pages opened at a person’s request |
| Google-Extended | Control token | Controls Gemini use of your content, not Google Search | |
| Common Crawl | CCBot | Dataset | Opts out of a public web dataset that many models use |
Swipe sideways to see every column
Last reviewed October 2026. Companies add and rename crawlers, so check each company’s documentation before you change your rules.
How the AI crawler access checker in Ranklyst works
Access is part of about 30 measurable checks, so the same site gets the same result every time.
How to run an AI crawler test with Ranklyst
No setup, no account and no technical knowledge required.
- 1
Enter your website and email
Paste your URL and the email where you want your report. There is no account to create.
- 2
Ranklyst reads your robots.txt and reviews your pages
Ranklyst reads your public robots.txt file, flags rules that block AI crawlers, and reviews up to 12 of your most important pages as part of about 30 measurable checks.
- 3
Read your report and fix the blocks
Receive your GEO, SEO and AEO scores, a prioritized fix list with effort and impact, a PDF and a share link, usually within a few minutes.
How to check if AI crawlers can access your website, step by step
A real AI crawler test has three layers: your rules, your network and your logs. This process covers all three.
- 1
Open your robots.txt file
Go to yourdomain.com/robots.txt in a browser. Every subdomain needs its own file, so check blog.yourdomain.com and app.yourdomain.com separately if you use them.
- 2
Look for catch-all rules first
Find any block that starts with “User-agent: *”. If it contains “Disallow: /”, it blocks every crawler from the whole site. This rule often survives from a staging site and is the most common cause of access problems.
- 3
Search the file for each AI crawler
Look for the names in the crawler table on this page. Note which ones have a rule and which ones have none. A crawler with no rule follows your catch-all block.
- 4
Decide what you want for each type of bot
Treat training, search and user-triggered crawlers as three separate choices. If you want AI assistants to cite you, allow the search and user-triggered crawlers. Decide on training crawlers separately.
- 5
Edit the rules, one block per crawler
Robots.txt rules apply per user-agent, so write a separate block for each crawler you want to allow or block. Save the file, reload it in your browser, and confirm the change shows.
- 6
Check your firewall and CDN bot settings
Security tools can block bots even when robots.txt allows them. Open your CDN or firewall dashboard and look for bot protection rules that challenge or block unfamiliar crawlers.
- 7
Confirm in your server logs and test again
Look for the crawler names in your server logs to confirm real visits and successful responses. Repeat this check after every release, CDN change or robots.txt edit.
Should you allow or block AI crawlers? Start from your goal
There is no single right answer. The right setup depends on what you want from AI assistants.
| Your goal | Allow | Block | Note |
|---|---|---|---|
| I want AI assistants to cite my pages | Search and user-triggered crawlers | Nothing required | Check that your firewall does not challenge these bots. |
| I want citations but no model training | Search and user-triggered crawlers | Training crawlers | Block each training crawler by name. Blocking one name does not block them all. |
| I have paid or private content | Only public pages | Use a login, not robots.txt | Robots.txt is a request, not a lock. |
| I am not sure yet | Search crawlers first | Decide on training later | Review the choice every quarter as policies change. |
Swipe sideways to see every column
Blocking an AI crawler does not change how Googlebot ranks your pages. Google handles AI use of your content through a separate control, so your search rankings do not depend on these choices.
Five AI search problems that often start with crawler access
When something looks wrong in AI answers, match it to the likely cause before you change anything.
| What you see | Likely cause | Next step |
|---|---|---|
| An assistant says it cannot open your website | A robots.txt block, a firewall or CDN bot rule, or a login wall | Work through the manual steps above in order, then confirm in your logs. |
| An assistant describes an old version of your page | Its index has not refreshed, or a crawler could not fetch the new version | Confirm the search crawler can reach the page, then test again after a few weeks. |
| An assistant still knows things from pages you blocked | It learned them before you blocked, or from other sources about you | Blocking is not a retroactive delete. Keep your public facts accurate. |
| Assistants name your competitors and not you | Access is only one factor. Clarity, trust and answers matter too | Run an AI mention check and a GEO audit. |
| Your pages rank in Google but never show up in AI answers | Googlebot has access, but an AI crawler may not | Compare your AI crawler rules with your SEO basics using an SEO audit. |
Swipe sideways to see every column
A 12-point AI crawler access checklist you can use today
Keep this list next to your audit report. If you can tick every line, AI crawlers can reach what you want them to read.
- Your robots.txt file loads at the root of every domain and subdomain
- No “User-agent: *” rule blocks the whole site
- You know which AI crawlers have their own rule
- Search crawlers are allowed if you want citations
- You chose a training policy on purpose
- You wrote one block for each crawler
- Your rules match each company’s current documentation
- Your firewall does not challenge crawlers you allow
- Private content sits behind a login
- Your server logs show successful visits from allowed bots
- You re-check after every release or CDN change
- You review your policy every quarter
Six AI crawler mistakes that quietly close your site
These mistakes show up again and again, even on well-run sites.
What any AI crawler checker cannot tell you
Any tool that promises a full picture from outside your server is overselling. Here is what a robots.txt check, including ours, cannot show.
Where an AI crawler access check pays off fastest
Any site that wants to be found through AI assistants can benefit. These four types usually see the clearest gaps.
Free AI crawler tools and methods compared with Ranklyst
Each method covers a different layer. Here is where each one fits.
| Method | Best for | What it leaves out |
|---|---|---|
| Reading robots.txt by hand | Spotting blocks quickly at no cost | Easy to miss catch-all rules and says nothing about firewalls or logs |
| Server log analysis | Proof of which bots really visit and what they receive | Needs log access and skill, and only shows bots that already arrived |
| CDN and firewall bot dashboards | Seeing bot traffic and security rules in one place | Does not review your robots.txt policy or your site signals |
| Ranklyst | A fast, readable robots.txt review inside a full audit with a GEO score and ranked fixes | Cannot see firewall, CDN or server-log data |
Swipe sideways to see every column
Check AI crawler access for $2
One payment, one website, no subscription. Your report also includes SEO, AEO and GEO scores and an AI visibility test.
Ranklyst reads public pages only and follows your robots.txt file. About RanklystBot
AI crawler checker questions answered
Clear answers before you run your first check, in English, Spanish and Roman Urdu.
What is an AI crawler checker?
An AI crawler checker shows whether the bots that AI companies use can reach your website. It reads your robots.txt rules, looks for blocks that stop AI crawlers, and helps you fix them. Ranklyst includes this check in every audit.
What is an AI crawler?
An AI crawler is an automated bot run by an AI company that visits web pages. Some collect content to train models, some build search indexes for AI assistants, and some fetch a page when a person asks an assistant to open it. Examples include GPTBot, ClaudeBot and PerplexityBot.
How do I check if AI crawlers can access my website?
Open your robots.txt file and look for catch-all blocks and rules that name AI crawlers. Then check your firewall and CDN bot settings, and look for the crawler names in your server logs. A Ranklyst audit reads your robots.txt file and flags rules that block AI crawlers.
What is the difference between training, search and user-triggered AI crawlers?
Training crawlers collect content to build models. Search crawlers build the indexes that assistants use to find and cite pages. User-triggered crawlers fetch a page when a person asks an assistant to open it. You can make a separate choice for each type.
Which AI crawlers should I allow in robots.txt?
It depends on your goals. If you want assistants to cite your pages, allow the search and user-triggered crawlers. Decide on training crawlers separately, based on whether you want your content used to train models. Read each company’s documentation first, because names and roles change.
If I block GPTBot, does ChatGPT stop seeing my site?
Not necessarily. GPTBot is OpenAI’s training crawler. OpenAI uses other crawlers for search and for user requests, and they follow their own rules. The same pattern applies to other companies, so blocking one name rarely blocks everything.
Does blocking AI crawlers hurt my Google rankings?
No. Googlebot is separate from AI crawlers, and blocking an AI crawler does not change how Google ranks your pages. Google-Extended, a robots.txt control for Gemini training and grounding, does not affect Google Search either.
What is Google-Extended?
Google-Extended is a robots.txt token, not a separate crawler. It lets you control whether Google uses your content for its Gemini models. Blocking it does not remove your pages from Google Search.
Is there a free AI crawler checker?
You can check by hand for free. Read your robots.txt file, check your firewall settings and look at your server logs, using the steps on this page. Ranklyst costs $2 once and reads your robots.txt rules for you as part of a full audit, then ranks the fixes.
Is there a free AI website crawler I can use on my own site?
The term “AI crawler” also describes AI-powered tools that crawl sites for people. For your own site, Google Search Console is free and shows how Googlebot crawls it. Ranklyst reviews up to 12 key pages with about 30 checks, but it is not built to crawl thousands of pages.
How do I run an AI crawler test?
Test three layers. First, test your rules by reading your robots.txt file. Second, test your network by checking firewall and CDN bot settings. Third, test reality by looking for crawler names in your server logs. All three must agree before you can say a crawler has access.
Why does an AI assistant say it cannot access my website?
Common causes are a robots.txt rule that blocks the assistant’s crawler, a firewall or CDN rule that challenges bots, a page that needs a login, or a catch-all block left from staging. Check them in that order.
Do AI crawlers respect robots.txt?
Major AI companies document their crawlers and say they follow robots.txt, but treatment of user-triggered requests can differ, and some unidentified scrapers ignore it. Read each company’s documentation, and use firewall rules, not robots.txt, to stop bots that do not follow the rules.
Does robots.txt keep my content private?
No. Robots.txt is a request that well-behaved bots follow, not a lock. Put private or paid content behind a login. Never rely on robots.txt to protect it.
Do I need an llms.txt file for AI crawlers?
It is optional. The llms.txt file is a proposed convention, and no major AI provider has promised to use it. Open crawler access and clear pages matter far more, so fix your robots.txt rules first.
How often should I check AI crawler access?
Check after every release, redesign, CDN or firewall change, and robots.txt edit, and then about once a month. Staging rules and security settings change quietly, and a block can sit unnoticed for weeks.
What does Ranklyst check for AI crawler access?
Ranklyst reads your public robots.txt file and flags rules that block AI crawlers, including catch-all blocks. It reviews up to 12 of your important pages and includes the result in your GEO score. It cannot see firewall, CDN or server-log data, so confirm real visits in your logs.
How much does an AI crawler check cost?
A Ranklyst audit costs $2 per website, one time, with no subscription. The same report also includes your SEO, AEO and GEO scores and an AI visibility test.
¿Cómo comprobar si los rastreadores de IA pueden acceder a mi sitio web?
Abre tu archivo robots.txt y busca bloqueos generales y reglas que nombren rastreadores de IA. Después revisa la configuración de tu cortafuegos y de tu CDN, y busca los nombres de los rastreadores en los registros de tu servidor. Ranklyst lee tu robots.txt y señala las reglas que bloquean a los rastreadores de IA por un pago único de $2.
AI crawler checker kya hai?
AI crawler checker yeh dekhta hai ke AI companies ke bots aapki website tak pahunch sakte hain ya nahi. Yeh aapki robots.txt ke rules parhta hai, un rules ko dhoondta hai jo AI crawlers ko block karte hain, aur unhein theek karne mein madad deta hai. Ranklyst har audit mein yeh check shamil karta hai aur sirf $2 mein poori report deta hai.
Find out if AI bots can read your website
Run your Ranklyst audit and see whether your robots.txt blocks AI crawlers, what to fix first, and how AI assistants treat your site.
One-time payment · No account · Report in a few minutes