As of October 2026, llms.txt is a proposed convention, not a standard, and Google's own documentation says you do not need special AI files to appear in its AI features. A small site gains more from letting the right crawlers in through robots.txt and publishing clear, answer-first pages than from adding llms.txt.
Key takeaways
- llms.txt is a Markdown file proposed in 2024 to help AI tools understand a site. It is not an official web standard.
- Google says there are no additional requirements, special files or special structured data needed to appear in AI Overviews or AI Mode.
- Robots.txt controls which crawlers may fetch your pages. It is voluntary, and it does not control how an AI describes your business.
- Blocking a training crawler and allowing a search crawler are separate decisions, and the major vendors document them separately.
- Nobody can promise that any file or setting will get you recommended by an AI assistant. Be wary of anyone who does.
What is llms.txt, and does it help a small business?
The llms.txt idea was proposed by Jeremy Howard in 2024. It is a plain Markdown file placed at the root of a site, usually at /llms.txt, containing a site name, a short summary and a list of links to your most useful pages, so an AI agent can find the important material without wading through menus and scripts. The proposal says its purpose is to help agents use a website at the moment someone asks a question.
The honest answer on benefits is that it is unproven. As of October 2026, Google's documentation on AI features states: "You don't need to create new machine readable files, AI text files, or markup to appear in these features." The same page says there are no additional requirements or special optimizations needed to appear in AI Overviews or AI Mode, and that normal SEO best practices still apply. We looked for statements from OpenAI, Anthropic and Perplexity describing llms.txt as something their crawlers or assistants require or use for ranking, and did not find any in their crawler documentation. That is not proof that nobody reads the file, only that we could not verify any benefit.
So should you add one? If your platform makes it easy and you have a tidy list of key pages, it costs little and is unlikely to hurt. Do not pay for it as a service, do not expect it to raise your visibility, and do not let it replace the basics. Some platforms, including Shopify, may not let you place a file at the root of your domain, so check feasibility before spending time.
What does robots.txt actually control?
Robots.txt is a text file at the root of your site that tells crawlers which URLs they may fetch. Google describes it as mainly a way to avoid overloading your site with requests. It has limits worth knowing:
- It cannot enforce anything. Google notes it is up to the crawler to obey it.
- It is not a way to keep a page out of Google. A blocked URL can still be indexed if other sites link to it; use a noindex tag or a password instead.
- It controls crawling, not what an assistant says about you from other sources it already knows.
On Shopify, every store has a default robots.txt, and you can customize it by adding a robots.txt.liquid template. Shopify calls that an unsupported customization, and warns that incorrect use can cause a loss of all traffic. Our advice: do not edit it unless you understand each line, and test afterward.
Which AI crawlers exist, and how do I control them?
This table reflects each vendor's own documentation as of October 2026. Bot names and behavior change, so check the vendor pages before editing anything.
| Crawler | Operator | Purpose | How to control |
|---|---|---|---|
| GPTBot | OpenAI | Crawls content that may be used to train generative AI models | Disallow in robots.txt to signal your content should not be used for training |
| OAI-SearchBot | OpenAI | Powers search features in ChatGPT | Allow or disallow in robots.txt. OpenAI says changes can take about 24 hours to take effect, and sites that opt out will not appear in ChatGPT search answers |
| ChatGPT-User | OpenAI | Visits pages when a user asks ChatGPT to | OpenAI says robots.txt rules do not apply because the visit is user-initiated |
| ClaudeBot | Anthropic | Collects web content that helps improve its models | Disallow for User-agent: ClaudeBot in robots.txt; Anthropic also supports Crawl-delay |
| Claude-SearchBot and Claude-User | Anthropic | Search result quality, and retrieving pages when a user asks | Blocking each in robots.txt stops that type of access |
| PerplexityBot | Perplexity | Surfaces and links sites in Perplexity search; not used for foundation model training | Allow it in robots.txt to be listed; Perplexity publishes IP ranges |
| Perplexity-User | Perplexity | Fetches pages when a user asks a question | Perplexity says it generally ignores robots.txt because a person triggered the visit |
| Google-Extended | Controls use of content for training future Gemini models and for grounding | A robots.txt token. Google says it does not affect inclusion in Google Search and is not a ranking signal |
Notice what is missing: Google's AI Overviews and AI Mode are part of Search, so Google says the existing robots.txt and snippet controls apply, and Google-Extended is a separate choice about Gemini. If you want to appear in Google Search, do not block Googlebot.
What low-cost steps are worth doing now?
- Open your robots.txt. Visit yourdomain.com/robots.txt and read it. Make sure you are not blocking Googlebot or search crawlers by accident.
- Decide your training stance. Some owners are happy for content to be used for model training, others are not. Both are legitimate. Decide, then set GPTBot, ClaudeBot and Google-Extended accordingly.
- Keep search and retrieval bots allowed if you want to be found. Blocking OAI-SearchBot, for example, opts you out of ChatGPT search answers.
- Make pages readable as plain HTML. Put key facts such as services, prices, hours and address in real text, not only in images or scripts.
- Write answer-first pages. Open with a direct answer, then support it. Our guide on answer engine optimization shows the pattern.
- Fix your listings. A complete Google Business Profile (see our Google Business Profile setup and optimization service) and consistent business details give every engine more to work with.
- Treat llms.txt as optional. If you add one, keep it short and accurate and recheck it when your pages change.
None of this is a sure route to a recommendation. AI assistants draw on many sources, and results vary by question and by day. For the wider picture, read how to get found on ChatGPT and AI search.
Should I block AI crawlers to protect my content?
That is a business decision, and the trade-off is real. Blocking training crawlers signals that you do not want your content used to train models, but it does not remove what is already collected, and since robots.txt is voluntary, compliant bots follow it and others may not. Blocking search or retrieval bots can make you invisible in those assistants' answers. For a local service business that wants new customers, we usually suggest blocking nothing you do not have a reason to block, and revisiting the choice as the vendor policies change.
If you do not run the site yourself, ask whoever does to show you the current robots.txt and explain every rule. If you want help making a Fort Myers or Southwest Florida business easier for AI assistants to read, see our SEO and AI search in Fort Myers page.
Frequently asked questions
Does llms.txt improve my Google rankings?
There is no evidence that it does. Google's documentation on AI features says you do not need to create new machine readable files, AI text files or markup to appear in AI Overviews or AI Mode, and that normal SEO practices remain the foundation. As of October 2026 we could not verify any search engine or assistant documenting llms.txt as a ranking input, so treat it as optional and unproven.
Is llms.txt an official standard?
No. It is a proposal published in 2024 by Jeremy Howard, describing a Markdown file at the root of a site that lists a summary and key links for AI agents. It has not been adopted as an internet standard, and different tools may or may not read it. Because the situation changes, check current vendor documentation before relying on it, and do not buy it as a paid service.
What is the difference between GPTBot and OAI-SearchBot?
They do different jobs. OpenAI says GPTBot crawls content that may be used to train its generative AI models, while OAI-SearchBot powers search features in ChatGPT and surfaces websites in results. Each is controlled separately in robots.txt, so you can disallow one and allow the other. OpenAI notes sites that opt out of OAI-SearchBot will not appear in ChatGPT search answers, and changes can take about 24 hours.
Will blocking Google-Extended remove me from Google Search?
No. Google says Google-Extended does not affect a site's inclusion in Google Search and is not a ranking signal. It is a separate robots.txt token that controls whether content may be used for training future Gemini models and for grounding. Blocking Googlebot is a different matter and would remove your pages from Search, so be careful which line you edit.
Can I stop AI tools from reading my website at all?
Not completely. Robots.txt is voluntary, so well-behaved crawlers follow it and others may not. Some user-triggered fetchers, such as ChatGPT-User and Perplexity-User, are documented by their vendors as not governed by robots.txt because a person requested the visit. For stronger control you would need measures such as password protection or server-level rules, which also block real visitors, so most small businesses should not go that far.
Do I need to do anything about AI crawlers if I have a small local business?
Do a basic check, not a project. Confirm your robots.txt is not blocking search crawlers by mistake, decide whether you are comfortable with training crawlers, and keep the bots you want for discovery allowed. Then put your effort into clear pages, accurate business details and a strong Google Business Profile, which help in every kind of search. Revisit your settings once or twice a year.
Sources
- Google Search Central: AI features and your website
- Google crawling documentation: Google common crawlers (Google-Extended)
- Google Search Central: Introduction to robots.txt
Vendor crawler details come from OpenAI's, Anthropic's and Perplexity's own bot documentation and the llmstxt.org proposal, read in October 2026. Check those pages for current names and rules.
Our Local + AI (GEO) Optimization service at $480 focuses on the fundamentals: answer-first page structure, FAQ markup and local schema. We do not sell llms.txt as a ranking tactic, and we promise no placement in any AI assistant. A person reviews every page. For a wider look at what you may be missing, ask about the free AI visibility report.
