llms.txt vs robots.txt for Ecommerce: What Does Each File Actually Do?
robots.txt and llms.txt solve different website problems. robots.txt communicates crawler access rules. llms.txt is a voluntary convention for giving AI agents concise guidance and links to selected public content. Neither file fixes weak product information, guarantees AI visibility, or determines whether a product should be recommended.
Tarun Babbar
In the llms.txt vs robots.txt choice, robots.txt is the file that matters for an online store: it tells each crawler which pages it may fetch, and blocking OAI-SearchBot keeps your pages out of ChatGPT search answers. llms.txt controls nothing, neither Google, OpenAI nor Anthropic documents it as an input, and four of the five stores in our sample served their platform's default file without writing one.
A short llms.txt costs little and has no documented downside; robots.txt is where a mistake costs you answers, so check it first. Three questions sort most cases: is the problem permission (robots.txt), a summary for agents (llms.txt), or a knowledge gap, where the AI lacks the facts it needs about your product (neither file)?
Key takeaways
robots.txt sets access per crawler and per path, and search, user and training bots each have their own name.
Shopify serves a default llms.txt for every store, so most Shopify teams have one without trying.
In our scans of six stores, the one robots.txt fault was a malformed file that left its rules unreadable; none blocked an AI search bot.
What is the difference between llms.txt and robots.txt?
robots.txt is a plain-text file of rules at your domain's root. Google describes it as a file that "tells search engine crawlers which URLs the crawler can access on your site," and the rules have been an IETF standard since September 2022 (RFC 9309). llms.txt is a Markdown file that summarizes a site and links to its most useful pages for language models. Its author published it as a proposal in September 2024 and revised it in August 2026 (llmstxt.org).
robots.txt | llms.txt | |
|---|---|---|
Status | IETF standard (RFC 9309) | Proposal, version 2 |
What it controls | Which crawler may fetch which path | Nothing; it is guidance only |
Who documents using it | Google, OpenAI, Anthropic and Perplexity all name their crawlers' robots.txt tokens | Neither Google Search, OpenAI nor Anthropic; Chrome's experimental Lighthouse audit only checks that one exists |
Cost of getting it wrong | A blocked search bot drops your pages from that engine's answers | None documented |
On Shopify | Editable through the theme's robots.txt.liquid template | A default file mirrors /agents.md unless the theme overrides it |
As of October 2026. Sources: Google Search Central, Google AI features guide, Chrome Lighthouse, Shopify robots.txt.liquid, Shopify changelog.
Which AI crawlers should an online store allow?
Allow the search bots and user-triggered fetchers, and treat training bots as a separate business decision. Each name controls a different outcome.
Crawler | Company | What it does | What blocking it does |
|---|---|---|---|
OAI-SearchBot | OpenAI | Surfaces sites in ChatGPT search | Pages leave ChatGPT search answers, though they can still appear as navigational links |
GPTBot | OpenAI | Collects content for model training | Opts your content out of training |
ChatGPT-User | OpenAI | Fetches a page when a user asks | OpenAI says robots.txt rules "may not apply" to these user actions |
Claude-SearchBot, Claude-User, ClaudeBot | Anthropic | Search indexing, user fetches, training | Blocking the first two may reduce your visibility in Claude |
PerplexityBot, Perplexity-User | Perplexity | Search results; user fetches | PerplexityBot leaves Perplexity search; Perplexity-User "generally ignores robots.txt rules" |
Googlebot | Crawls for Google Search, whose AI Overviews and AI Mode use only indexed pages | Pages can still be listed, but without a snippet, so they cannot appear in AI Overviews or AI Mode | |
Google-Extended | A token for Gemini training and grounding | No effect on Google Search, which includes AI Overviews and AI Mode |
As of October 2026. Sources: OpenAI, Anthropic, Perplexity, Google, Google AI features guide.
Consider a fictional furniture store with 4,000 products and tens of thousands of filtered collection URLs. It wants AI search bots on product pages but not on filters, and its legal team has opted out of model training. Its file could read:
User-agent: OAI-SearchBot
User-agent: Claude-SearchBot
User-agent: PerplexityBot
Disallow: /cart
Disallow: /checkout
Disallow: /*?*filter
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
Disallow: /
User-agent: *
Disallow: /cart
Disallow: /checkout
Disallow: /*?*filter
Sitemap: https://www.example-furniture.com/sitemap.xml
Note the repeats. Under the standard, a crawler follows only the group that names it and falls back to the * group only when no group matches, so a named bot ignores everything written for *. A Shopify team that adds one AI bot group to the theme's robots.txt can reopen the default cart, checkout and filter blocks to that bot by accident. Blocking Google-Extended also keeps your pages out of Gemini's grounding, so a store that wants Gemini to cite it should drop that token from the training group. OpenAI and Perplexity say changes take about a day to reach their search results.
Is llms.txt worth it for an online store?
Only as a cheap default, since Google says you don't need "new machine readable files, AI text files, markup, or Markdown to appear in Google Search" (Google Search Central), and OpenAI's publisher FAQ on ChatGPT search does not mention the file (OpenAI Help Center). One Google tool does check for it: Lighthouse's Agentic Browsing category, which is experimental and needs Chrome 150 or later.
Most Shopify stores already have one. Shopify manages a default agents.md file for every store and points /llms.txt and /llms-full.txt to the same content unless the theme adds its own template, as its May 2026 changelog explains (Shopify). Four of the five stores in our sample served their platform's default, about 4.5 KB each; the fifth wrote its own, over 100 KB.
How we measured this. Five online brands in beauty, jewelry and home goods, scored between October 2 and 8, 2026, on 20 shopper questions each (30 to 32 of each brand's 40 answers came from questions that left the brand out), answered once by the consumer versions of ChatGPT and Google AI Mode. File and robots.txt checks come from the technical scan run with each scorecard. Across the five, 75 of 158 answers to unbranded questions (47%) named the brand, and the custom-file brand was named least often. Five brands cannot show cause. The sixth store below had a technical scan only, on October 22.
We think llms.txt spreads because it is easy to tick off, and box-ticking is how stores end up running the same SEO-driven playbook with similar results. Keep your platform's default, or write a short file that links your shipping, returns, sizing and warranty pages, and spend the saved hours on the product facts shoppers ask about.
What goes wrong with robots.txt on store sites?
The one fault we found was accidental. None of the five stores above blocked a search or training bot.
A sixth brand, a beauty company outside that sample, had a different problem when we scanned it in October 2026. Every group in its file, including the * group that held the cart, checkout and search rules, opened with a doubled directive such as User-agent: User-agent: *, its Sitemap line had been pasted onto the end of a Disallow rule, and our scanner could not resolve a verdict for any crawler. RFC 9309 says that when no group names a crawler and no valid * group exists, no rules apply, so those Disallow rules most likely reach no crawler, and the file no longer declares a sitemap.
Access can also fail outside the file: a CDN bot rule can refuse a crawler that robots.txt allows. Cloudflare, for one, announced in July 2025 that it would block AI bots by default on the sites it hosts (MIT Technology Review). And robots.txt is no lock: RFC 9309 states the rules "are not a form of access authorization," so private pages need a login, not a Disallow line.
Run this order of checks:
What you see | Likely cause | First check | Owner |
|---|---|---|---|
Almost no ChatGPT referrals, pages rank in Google | OAI-SearchBot blocked in robots.txt or at the CDN | Your robots.txt group for OAI-SearchBot; CDN bot settings | Developer |
Search Console lists robots.txt errors | Malformed lines, so rules are skipped | Search Console's robots.txt report | Developer |
AI bots crawl cart, checkout or filter URLs | A named bot group without the | Compare each named group with the | Developer or SEO lead |
Legal wants no model training, marketing wants AI search | One rule set for all bots | Separate groups for training and search bots | Legal and ecommerce leads |
What can neither file do?
Neither file adds a fact to a product page or makes an engine recommend you. robots.txt only removes a barrier, so an allowed page can still go uncrawled and uncited. llms.txt only points, and it cannot open a page that robots.txt blocks. Product facts and markup are a separate job, covered in our guide to product schema.
How does Gravton check crawler access for AI shopping?
A robots.txt check tells you whether a crawler may enter, not whether your products show up. Gravton's free audit tests the second: a Demand Snapshot of the shopper questions you miss across the AI engines your buyers use, who wins them, and ranked fixes from access problems to missing product facts. Site Health checks whether AI can reach, read, understand, and trust your product and category pages.
Get your free AI visibility audit to find out whether access is your problem at all.
Frequently asked questions
Will blocking GPTBot remove my products from ChatGPT shopping?
No. GPTBot collects content for training OpenAI's models, while appearance in ChatGPT search answers depends on OAI-SearchBot, a separate rule. A store can opt out of training and stay in ChatGPT search by disallowing GPTBot and allowing OAI-SearchBot, as long as no CDN or firewall rule blocks the second.
Can robots.txt keep my checkout and account pages private?
No. robots.txt asks crawlers to stay away, and well-behaved ones do, but it is not access control, and some user-triggered fetchers ignore it. Google adds that the file is not a way to keep a page out of its results. Protect account and order pages with a login, and use a noindex tag for pages you want kept out of search.
Should llms.txt list every product in the catalog?
No. A catalog changes daily, and a long file goes stale fast. Your sitemap and product feeds already carry every product URL, with live price and stock in the feeds. Keep llms.txt short: what the store sells, plus links to category, shipping, returns and sizing pages. Version 2 of the proposal also lets each section of a site have its own file.
How do I check what crawlers can actually read on my site?
Start with Search Console's robots.txt report, which shows the files Google found, when it last fetched them, and errors that stop a rule from being used. Then check a product, a collection and a checkout URL against each AI bot's group, and ask whether a CDN rule treats AI crawlers differently.
What should you read next?
Technical GEO and AI Crawlability for Ecommerce covers rendering, linking and faceted URLs.
Does Product Schema Help AI Shopping? covers Product and Offer markup and what it cannot fix.
What Is Generative Engine Optimization for Ecommerce? explains how an AI answer gets from question to buy link.
How Do I Track AI-Shopping Traffic? covers the analytics once crawlers can reach you.
Sources
Google Search Central. "Introduction to robots.txt." Accessed October 23, 2026.
IETF. "RFC 9309: Robots Exclusion Protocol." September 2022.
llmstxt.org. "Changes." Version 2, August 2026.
Google Search Central. "Optimizing your website for generative AI features on Google Search." Updated July 10, 2026.
Chrome for Developers. "Lighthouse agentic browsing scoring." Updated May 5, 2026.
Shopify. "agents.md template" and "Customize llms.txt, llms-full.txt and agents.md." May 28, 2026.
OpenAI. "Overview of OpenAI crawlers." Accessed October 23, 2026.
OpenAI Help Center. "Publishers and Developers FAQ." Accessed October 23, 2026.
Anthropic. "Does Anthropic crawl data from the web?" April 2026.
Perplexity. "Perplexity crawlers." Accessed October 23, 2026.
Google for Developers. "Google's common crawlers." Updated October 15, 2026.
MIT Technology Review. "Cloudflare will now, by default, block AI bots from crawling its clients' websites." July 1, 2025.
Google Search Console Help. "robots.txt report." Accessed October 23, 2026.
Gravton Labs. Pooled scorecard data and technical checks for five ecommerce brands, October 2026; technical scan of one beauty brand, October 2026. Internal datasets, anonymized.
Related resources
Continue with these related Gravton resources:
