robots.txt guide for AI crawlers

Part of the Agent Ready guides. Plain advice, no signup needed.

robots.txt is a small file at your domain root that tells crawlers which pages they may visit. AI shopping crawlers obey it, including GPTBot, Claude-Bot, and PerplexityBot.

The rule that matters

One blocked crawler means one assistant that never sees your store. The common failure is a copied template with "Disallow: /" for a bot the merchant has never heard of.

Minimal safe file

User-agent: GPTBot
Allow: /

User-agent: Claude-Bot
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: *
Allow: /

On Shopify

Edit robots.txt.liquid under Templates in the theme editor. On WooCommerce, use your SEO plugin file editor or edit the file at the site root.

Audit your store

More guides: ChatGPT Shopping Shopify schema robots.txt for AI