Search is splitting in two. Alongside the classic blue links, people now get answers straight from ChatGPT, Perplexity, Google AI Overviews and Claude — and those engines can only cite sites their crawlers are allowed to fetch and able to read. Paste a URL and this tool checks, in seconds, which AI crawlers you allow or block, whether your llms.txt and schema are in place, and how your homepage scores for AI search readiness.
For twenty years, SEO meant earning a place among ten blue links. That is no longer the whole game. A growing share of searches end inside an AI answer — ChatGPT, Perplexity, Google's AI Overviews, Microsoft Copilot, Claude — where the engine reads a handful of sources and writes the reply itself. The work of getting picked as one of those sources has a name now, Generative Engine Optimization, and it rests on a simple foundation: an engine can only cite a page it is allowed to fetch and able to read. This checker tests that foundation.
1. Can the AI crawlers reach you? Every major AI engine fetches the web with a named user-agent — OpenAI's GPTBot and OAI-SearchBot, Anthropic's ClaudeBot, PerplexityBot, Google-Extended for Gemini, and plain Googlebot for AI Overviews. A single line in robots.txt can quietly block any of them. Plenty of sites disallow these bots without realising they have just opted out of AI search entirely. The tool reads your robots.txt and tells you, bot by bot, who is allowed in and who is shut out.
2. Is your content actually in the page? Most AI crawlers do not run JavaScript. If your homepage ships an empty shell and paints its content in the browser, those crawlers see almost nothing — no matter how good the content is. The tool measures how much readable text exists in the raw HTML and flags pages that depend on JavaScript to show anything.
3. Can engines make sense of it? Structured data (Schema.org), a real title and meta description, and a clean H1/H2 outline all help an engine understand what a page is about and pull a clean answer out of it. The tool checks for each and, separately, looks for an llms.txt file — an optional, emerging convention for pointing AI tools at your key URLs.
You will see llms.txt mentioned everywhere right now, so here is the honest version. It is a small Markdown file that curates which pages on your site matter most to AI tools. Some tools read it; Google has publicly said it does not, and after eighteen months adoption sits at roughly one site in ten. We report whether you have one because it is cheap to add and harmless to have — but do not mistake it for the main event. Letting the crawlers in, serving content without a JavaScript wall, and marking pages up with schema will move your AI visibility far more than any llms.txt ever will.
If answer engines are blocked, open your robots.txt and review the rules for the bots flagged above — unblocking them is usually a one-line change. If the homepage came back as JavaScript-dependent, make sure your important content is server-rendered or pre-rendered so a crawler that does not execute JS still sees it. If schema or headings are missing, that is ordinary on-page SEO work the free website crawler can map across your whole site. For the broader picture, the robots.txt tester and the browser-based crawler pair naturally with this check.
It looks at whether AI search engines like ChatGPT, Perplexity, Google AI Overviews and Claude can actually reach and understand your site. That comes down to three things: whether your robots.txt lets their crawlers in, whether your main content is in the HTML (not hidden behind JavaScript), and whether you give engines clean structure and schema to work with.
The most common reason is a robots.txt rule that blocks their user-agents — GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot or Google-Extended. If a crawler is disallowed, that engine cannot fetch your pages, so it cannot cite you. The second most common reason is a site that renders its content with JavaScript, since most AI crawlers do not run JS and see a near-empty page.
It is optional. llms.txt is an emerging convention for telling AI tools which URLs matter most. Some tools read it; Google has said it does not use it, and adoption is still around one in ten sites. We report whether you have one, but a missing llms.txt is not a real problem — getting your robots.txt, rendering and schema right matters far more.
No. This is an instant single-page check of your homepage, robots.txt and llms.txt. The full crawler walks your whole site and reports broken links, redirects, status codes and metadata across every page. Run this for a quick AI-visibility read, then run a full crawl for the technical detail.
It is a transparent 0-100 score: 40 points for letting answer engines crawl you, 20 for having content in the HTML, 15 for structured data, 15 for a clear title/description/heading structure, and 10 for an optional llms.txt. Every result page shows the exact breakdown so nothing is a black box.