Free robots.txt checker

Enter a store URL. We fetch its robots.txt file and report whether AI shopping crawlers can reach the store.

We fetch one file: /robots.txt. Nothing else on the site is touched.

What this check looks at

Whether robots.txt allows known AI crawlers such as GPTBot, Claude-Bot, and PerplexityBot.
Whether a Cloudflare managed block is present in the served file.
Whether the file names key AI crawlers explicitly, so access does not rest on a wildcard.

Questions about these results? Contact waymarkhq@proton.me.