AgentorayBot

What it is

Agentoray audits how AI shopping agents — ChatGPT, Perplexity, Claude, Gemini and others — are able to read an e-commerce store. AgentorayBot is the fetcher that does that reading. It exists so a merchant can see their storefront the way those agents see it, and fix what they cannot read.

When it visits

Only when somebody asks it to. There is no background crawl and no discovery queue. A scan runs because a person entered that store's address on agentoray.com, or because the store's owner turned on weekly monitoring for it. If you have never seen AgentorayBot before and see it once, somebody ran a single scan.

What it requests

One scan reads a small, fixed set of things, and then stops:

• the homepage
• /robots.txt and any sitemaps it points to
• up to five product pages, found from your sitemap, your product feed, or by following links from the homepage
• a product feed if the platform publishes one — /products.json on Shopify, the Store API on WooCommerce
• shipping and returns policy pages, when they are linked from the footer
• /llms.txt and /agents.md if they exist

How politely

• At most 25 requests per scan, then the scan stops whether or not it has everything.
• 10 second timeout per request, at most five redirects, and no more than 5 MB read from any single response.
• One backoff on a 429. We honour Retry-After and try once more, then give up and record the rate limit as the result.
• GET only. It never submits a form, never signs in, never adds to a cart and never completes a checkout.
• One page per scan may be rendered in a headless browser, to check whether your product details are visible without JavaScript. That render loads the page the way a browser would; it does not click anything.

robots.txt

AgentorayBot reads your robots.txt — analysing it is one of the main things a report is for — but it does not currently apply your Disallow rules to its own requests. That is a deliberate choice and worth explaining rather than burying.

The single most valuable finding this product produces is “your robots.txt blocks the agents that answer shopping questions”. A store that disallows crawlers is exactly the store whose owner most needs to hear it. If we obeyed the rule, we could not report the problem — the scan would come back empty and the owner would learn nothing.

The trade we make in exchange: scans are always requested by a person, capped at 25 requests, never repeated automatically except on a weekly schedule the store's own watcher turned on, and every request says who it is. If you would rather we did not fetch your store at all, blocking us is a single rule — see below — and we will not work around it.

Blocking it

Block the user agent at your CDN or WAF. On Cloudflare, a WAF custom rule with the expression below is enough:

If you block us, a scan of your store will report that it was blocked, and score it the way any agent-blocked store scores — which is the honest outcome, because that is what an AI agent would also hit.

If something looks wrong — a scan you did not expect, a rate that hurt, anything else — email [email protected] and we will answer. There is a person on the other end.

What we keep

The results of a scan: check outcomes, evidence snippets used to justify them, and the product URLs we read. We do not keep full copies of your pages. The privacy policy covers the detail, including how long a report lives and how to have one taken down.