TOOL — FREE FROM EGGKNITE
AI Crawler Access Test
See whether AI and search crawlers can actually read your page. We request it as each crawler, compare every answer with a normal browser visit, and check what your robots.txt allows for each one.
From the Apex (AEO program) toolset.
We request the page once as a normal browser and once as each crawler, from our server. Read-only, one request each.
How it works
- A normal browser visit first. Our server requests your page the way Chrome would. Every crawler answer is compared with this one.
- Then one request per crawler. We send the user agent each company publishes for its bot: OpenAI, Perplexity, Google, Microsoft, Apple, Meta and Common Crawl.
- We compare the answers. For each bot you see the status code, the time, the size, and whether a bot check page came back in place of your content.
- We read your robots.txt once. For every crawler name, including Anthropic's bots and the robots.txt-only names Google-Extended and Applebot-Extended, we work out what the file allows on this exact URL.
What this test cannot see
- Our requests come from our own server. Each company's real crawler uses its own addresses, and a site that checks crawlers by IP address can answer it differently, so your server logs have the final word.
- Anthropic publishes robots.txt names for ClaudeBot, Claude-SearchBot and Claude-User but no full user agent strings, so for those we check robots.txt only.
- We read up to 512 KB of each response, follow up to 3 redirects and wait up to 8 seconds per request.
Let's grow.
EGGKNITE© 2026 ALL RIGHTS RESERVED — A VERTEX INNOVATION COLLECTIVE COMPANY