llms.txt, AI crawlers and structured data

llms.txt, AI crawler rules and structured data decide whether AI tools can read your site. They do not decide whether those tools recommend you. Google says no special AI files are needed for its AI features. Get the basics right, check your crawler rules, add the files that are cheap, and spend the rest of the effort elsewhere.

Readiness is not visibility

Two questions get blurred in most of what is sold as “AI SEO”.

• Readiness: can ChatGPT, Perplexity and Gemini fetch and understand your pages? This is technical and mostly binary. A blocked crawler or a page that only renders in JavaScript is a hard stop.
• Visibility: when a buyer asks about your category, are you named? This depends on what credible sources say about you, and it is measured by sampling answers, as set out in how to check whether ChatGPT recommends your business.

Readiness is a precondition. Fixing it removes a reason to be left out. It does not give the engines a reason to put you in. Any tool or score that reports one as the other is measuring the wrong thing.

What Google says

Google’s guidance on AI features and your website is short and worth reading in full. As of October 2026 it says:

• To appear as a link in AI Overviews or AI Mode, a page must be indexed and eligible to show a snippet in Search. There are no additional technical requirements.
• No special AI text files and no special schema.org markup are needed.
• Structured data should match the visible text on the page.
• The controls are the existing ones: nosnippet, data-nosnippet, max-snippet and noindex.

One trade-off to know before you reach for those controls: the snippet controls that keep a page out of AI answers also strip its snippet from ordinary results. For a business that wants to be found, they are rarely the right call.

AI crawler rules

This is where readiness most often fails, usually by accident: a security plugin or a CDN setting that blocks “bots” in bulk. Each engine documents its crawlers, and the split that matters is training versus search.

• OpenAI: its crawler documentation lists GPTBot for training, OAI-SearchBot for ChatGPT search, and ChatGPT-User for fetches a person triggers. Sites that block OAI-SearchBot are not shown in ChatGPT search answers. You can block GPTBot and still allow OAI-SearchBot.
• Perplexity: its crawler page lists PerplexityBot, which builds its index and follows robots.txt, and Perplexity-User for user-triggered fetches, which it says generally ignores robots.txt.
• Google: Google-Extended is a robots.txt token, not a separate crawler. It controls use of your content for Gemini training and grounding outside Search. Google’s crawler list says it does not affect inclusion or ranking in Google Search, so blocking it does not remove you from AI Overviews.

The practical check takes five minutes. Open yoursite.com/robots.txt, look for any rule naming these agents or disallowing everything, and check your CDN or firewall for bot blocking. Decide on training separately from search. Most B2B firms that want to be found should allow the search crawlers at the very least.

llms.txt

llms.txt is a proposal Jeremy Howard of Answer.AI published in September 2024: a Markdown file at the root of a site with a short summary and links to the pages that matter. It was designed for tools that need a clean map of a site, such as coding assistants reading documentation.

It is a community proposal, not a standard. Google’s guidance says it is not needed for AI features in Search. OpenAI and Perplexity document their crawlers and robots.txt, not llms.txt. Some third-party scores count the file as an “AI signal”, which says more about what is easy to detect than about what earns a recommendation.

My position: add one, because it takes under an hour and does no harm. zebranorth.com has one, listing the offers, the free tools, the actions a visitor can take and the main resources. Keep it to real pages, keep it in step with the sitemap, and do not expect it to move a visibility number.

Structured data

Structured data helps search engines understand what a page is: an organisation, an article, a service, a person. It does that for AI features too, because they sit on the same index. It is not an AI ranking lever.

The rule that matters is in Google’s structured data policies: the markup must describe what is visible on the page. Services the page does not mention, or a “founded” date nobody can check, fail that test. Star ratings for your own business marked up on your own site are treated as self-serving, and Google does not show them. Start with Organization on the home page, Article or BlogPosting on posts and Service on service pages, then test them in the Rich Results Test.

The order to do it in

1. Check the pages that sell are indexed and render without JavaScript.
2. Read robots.txt and the CDN settings; allow the search crawlers.
3. Make structured data true to each page. Remove anything it cannot back up.
4. Add llms.txt, listing real pages only.
5. Then measure visibility, and put the remaining effort into the sources the engines cite.

The honest limitation: none of this guarantees a mention. It removes the technical reasons to be left out. The commercial reasons, a vague category description or no one credible writing about you, are a positioning problem, and that is the work I do as a fractional CMO. For the search fundamentals underneath all of it, see B2B SEO that creates pipeline.

FAQ