Why Google Checks LLMs.txt While Search Ignores It

Google Search says it ignores LLMs.txt. Chrome Lighthouse now checks LLMs.txt in an experimental Agentic Browsing audit. Both statements are true, and neither means what the loudest LinkedIn post is about to tell you.

Here is the direct answer: Google Search does not use LLMs.txt for crawling, ranking, AI Overviews, or AI Mode visibility. Chrome Lighthouse checks how a site responds when its experimental agent-readiness audit requests /llms.txt. A missing file that returns a clean 404 is marked Not Applicable, not failed.

That distinction matters because business owners are being handed a familiar marketing shortcut: create one tiny file, become “AI ready,” and wait for machines to bring customers. It is an appealing story. It is also unsupported by Google Search’s own documentation.

The Contradiction Is Real. The Conclusion Usually Isn’t.

In June 2025, Google’s John Mueller wrote on Bluesky: “FWIW no AI system currently uses llms.txt.” He later pointed to server logs, saying consumer LLMs and chatbots would fetch normal pages for training or grounding but were not fetching the LLMs.txt file (https://bsky.app/profile/johnmu.com/post/3lrshm4gggs2v).

A year later, Chrome’s Lighthouse documentation added an experimental Agentic Browsing category. It evaluates WebMCP integration, accessibility for agents, layout stability, and LLMs.txt handling. Lighthouse 13.3.0, released May 7, 2026, added that category to its default configuration and was expected to ship with Chrome 150 (https://github.com/GoogleChrome/lighthouse/blob/main/changelog.md#1330-2026-05-07).

So did Google quietly reverse itself?

No. “Google” is doing two different jobs here:

• Google Search decides how content is discovered, indexed, ranked, and used in Search features such as AI Overviews and AI Mode.
• Chrome Lighthouse is a developer-audit tool. Its experimental Agentic Browsing category evaluates whether a page is technically prepared for emerging browser-agent interactions.

One product can ignore a file for Search while another product tests how a website responds when an agent-oriented tool requests it. That is not a secret ranking signal. It is two departments asking two different questions, which is less dramatic but considerably more useful.

What Google Search Actually Says About LLMs.txt

Google’s current optimization guide for generative AI Search features is unusually direct. Under its mythbusting section, it says you do not need to create new machine-readable files, AI text files, special markup, or Markdown to appear in Google Search or its generative AI capabilities. It specifically names LLMs.txt and says Google Search does not use it.

The guide goes further: maintaining an LLMs.txt file for other services is completely fine, but doing so will neither help nor harm visibility or rankings in Google Search because Google Search ignores it (https://developers.google.com/search/docs/fundamentals/ai-optimization-guide).

Google’s separate documentation for AI Overviews and AI Mode says the same thing from another direction: ordinary SEO best practices remain relevant, and there are no extra requirements or special optimizations needed to appear in those features (https://developers.google.com/search/docs/appearance/ai-features).

Those are the claims we can support. The claims we cannot support are:

• Google ranks sites higher because they publish LLMs.txt.
• LLMs.txt is required for Google AI Overviews or AI Mode.
• Passing the Lighthouse Agentic Browsing audit improves organic rankings.
• A Lighthouse warning proves that Googlebot or Gemini cannot understand the website.
• The existence of an audit means Google Search has adopted the proposal.

A developer tool checking a condition is not evidence that a ranking system uses that condition. Lighthouse also checks layout stability and accessibility because those qualities affect users and machine interaction. That does not turn every audit line into a new search-ranking lever.

What the Chrome Lighthouse Audit Really Checks

Chrome’s official Lighthouse documentation calls LLMs.txt an “emerging convention” for providing a machine-readable summary of a website’s content to LLMs and AI agents. The experimental Agentic Browsing category requires Chrome 150 or later, and its documentation explicitly says it is based on proposed standards rather than settled web requirements (https://developer.chrome.com/docs/lighthouse/agentic-browsing/scoring).

The most important detail is hidden in the audit behavior:

• If /llms.txt returns a server error, the audit flags it.
• If /llms.txt is missing and returns a normal 404, the audit is marked Not Applicable.
• Google’s documentation says providing the file is optional at the moment.

That means the screenshot warning “Fetch of llms.txt failed: Timed out fetching resource” does not prove the site failed because it lacked a file. It shows that Lighthouse tried to fetch the root path and got a timeout instead of a clean response. The failure is partly about reliability: an agent asked the server a simple question and the server did not answer cleanly.

Chrome’s recommended fix is to create a concise Markdown file at the website root, such as https://example.com/llms.txt, following the emerging specification (https://developer.chrome.com/docs/lighthouse/agentic-browsing/llms-txt). But Chrome also says a clean 404 is acceptable for this optional audit. That nuance is likely to get less engagement than “Google now requires LLMs.txt,” but it has the advantage of being true.

What LLMs.txt Was Designed to Do

LLMs.txt was proposed by Jeremy Howard in September 2024 as a standardized Markdown file placed at /llms.txt. The goal was to give LLMs concise context and selected links at inference time, especially where a full website is too large or cluttered to process efficiently (https://llmstxt.org/).

The proposal is closer to a curated orientation sheet than a permissions file.

A useful LLMs.txt file can:

• Summarize the site’s purpose in plain language.
• Point to important documentation, services, policies, or reference pages.
• Give an agent a smaller set of high-value links to inspect.
• Reduce navigation noise when a tool voluntarily chooses to request the file.

It does not control crawler permissions. That remains the job of robots.txt and related access controls. It is not a sitemap replacement, a structured-data replacement, or proof that the business deserves to be cited. It also cannot force an AI system to retrieve, trust, or use what it contains.

The key phrase is “when a tool voluntarily chooses to request the file.” A beautifully written LLMs.txt file sitting untouched at the root is still a beautifully written file sitting untouched at the root.

Why Lighthouse Can Care Even When Search Doesn’t

The apparent contradiction disappears when we separate search retrieval from browser-agent operation.

Google Search crawls the web through established systems and uses its normal index, ranking signals, content understanding, and Search controls. Its official guidance says LLMs.txt is unnecessary for this job.

A browser-based agent may operate differently. It could open a live website, inspect the page, use accessibility information, interact with forms, or discover machine-callable tools. For that task, a small orientation file might reduce the amount of wandering required before the agent understands the site’s high-level purpose.

This is why the Lighthouse category combines four ideas:

1. WebMCP integration: can the site expose structured tools or form capabilities to agents?
2. Agent accessibility: is the accessibility tree usable enough for machine interaction?
3. Layout stability: will shifting page elements make automated interaction unreliable?
4. Discoverability through LLMs.txt: can an agent retrieve a concise site summary if one is offered?

That is a browser-interaction checklist, not a Google Search recipe. Lighthouse itself says the category does not produce the usual weighted score from 0 to 100 because agentic-web standards are still emerging. It reports fractional pass/fail and informational signals instead.

Should Your Business Create an LLMs.txt File?

For many businesses, the answer is: yes, if it is inexpensive, accurate, and treated as an experiment—not as a ranking tactic.

A short, maintained file has limited downside. It is human-readable, easy to inspect, and may help tools that choose to support the convention. It can also force a useful internal exercise: deciding which pages best explain the business, its services, its proof, its policies, and the actions customers can take.

But it should not jump ahead of higher-value work.

Create or fix these first:

• Important service, product, and location pages that are crawlable and indexable.
• Clear explanations of who the business serves, what it does, and where it operates.
• Accurate contact, pricing, policy, availability, and service-area information.
• Reviews, case studies, credentials, original examples, and third-party proof.
• Consistent business details across the website, Google Business Profile, directories, and trusted platforms.
• Useful internal links, structured data where appropriate, and a clean XML sitemap.
• Stable pages, accessible forms, and obvious call, booking, quote, or purchase paths.

Then add LLMs.txt if you can keep it current.

Do not create one by dumping every URL on the site into a Markdown file. That recreates the navigation problem the proposal was meant to reduce. Curate the links. Explain the site briefly. Include only canonical, useful destinations. Avoid stuffing sales claims or keywords into it like a meta-keywords tag wearing an AI costume.

A Practical LLMs.txt Decision Test

Use this five-question filter:

1. Is the underlying website information trustworthy?

If service pages are vague, locations conflict, policies are outdated, or proof is missing, LLMs.txt will summarize confusion more efficiently. That is not a win.

2. Can the business maintain the file?

A stale orientation file can send agents toward retired products, outdated pricing, closed locations, or broken links. Assign an owner and update it when important site content changes.

3. Are you solving an observed problem?

Check server logs to see whether anything requests /llms.txt. Run the experimental Lighthouse audit. Test the response directly. If no tool retrieves it, record that honestly rather than reporting imaginary AI traffic.

4. Is the implementation technically clean?

The root URL should respond quickly, use a sensible plain-text or Markdown content type, contain valid concise Markdown, and link to canonical HTTPS pages. A timeout is worse evidence of agent readiness than a clean 404.

5. Are the fundamentals already funded?

If the choice is between creating LLMs.txt and fixing a broken quote form, fix the form. If it is between LLMs.txt and publishing a genuinely useful service guide backed by evidence, publish the guide. Customers cannot buy from a text-file trend.

What Evidence Would Prove Real Adoption?

The cleanest way to evaluate LLMs.txt is to stop treating announcements as usage evidence.

A real adoption claim should be supported by at least one of these:

• Official documentation from a crawler or agent saying it requests and processes /llms.txt.
• Source code or a published technical specification showing when the request occurs and how the response is used.
• Repeated server-log requests from an identifiable, verifiable user agent or IP range.
• Controlled testing showing that changing the file changes the system’s behavior while the underlying pages remain constant.

Lighthouse requesting the path proves only that Lighthouse requests the path during this audit. It does not prove that Googlebot, Gemini, AI Overviews, AI Mode, ChatGPT, Claude, or Perplexity does the same in ordinary operation.

Server logs are useful, but even they need interpretation. A request could come from an SEO scanner, plugin, researcher, security bot, or curious person—not an answer engine consuming the file. Verify the user agent and network identity where possible. Look for repeated behavior over time. Compare requests for LLMs.txt with requests for the pages it links to.

The most persuasive evidence would be a documented consumer explaining what it does with the file. Until then, “the file exists,” “a plugin generated it,” and “an audit checked it” are implementation facts, not outcome evidence.

What a Sensible LLMs.txt File Should Contain

If you decide to create one, keep it deliberately small. The proposal expects Markdown with a top-level heading, a concise description, and curated sections of important links. For a service business, that might include:

• The main services and the strongest canonical service pages.
• Locations or service areas with accurate dedicated pages.
• About, credentials, team, or methodology information.
• Case studies, customer evidence, or independently verifiable proof.
• Pricing guidance, policies, FAQs, and contact or booking instructions.
• Optional resources that are useful but not essential to understanding the business.

Do not copy the homepage, manufacture claims, or create AI-only facts that humans cannot verify elsewhere. LLMs.txt should point toward the same truthful source material the website presents to customers. If the file says you serve Calgary while the website says Grande Prairie, you have not optimized anything. You have created a faster route to inconsistency.

How to Run the Lighthouse Check Without Misreading It

Chrome’s current documentation says the experimental category requires Chrome 150 or later. In a supported version:

1. Open the page you want to test.
2. Open Chrome DevTools and select Lighthouse.
3. Choose Navigation mode and the Agentic Browsing category.
4. Run the analysis and open each warning for its detailed reason.
5. Test https://yourdomain.com/llms.txt directly in a separate tab.

Interpret the result carefully:

• 200 with valid content: the file exists and was retrievable. This does not prove any crawler uses it.
• 404 and Not Applicable: the optional file is absent, but the server responded normally.
• Timeout or 5xx error: investigate hosting, CDN, firewall, routing, or deployment behavior.
• Malformed content warning: fix the file if you intend to offer it; otherwise decide whether a clean 404 is more honest.

Also review the other Agentic Browsing findings. An inaccessible form or unstable layout can prevent a live agent from completing a task even if LLMs.txt is perfect. The tiny text file should not become the valedictorian of a failing website.

The Bigger Lesson: Google Is Not One Opinion

Marketers often say “Google says” as if every Google product, team, experiment, patent, talk, and developer tool belongs to one synchronized mind. It does not.

Search Central documentation explains what Google Search uses. Chrome documentation explains what browser developers are experimenting with. Lighthouse gives deterministic technical feedback. A Chrome experiment can be worth watching without becoming a Search requirement. A Search team statement can remain correct while a browser team explores a different use case.

This distinction will matter more as agentic browsing develops. Websites may need to support two related but separate goals:

• Recommendation readiness: being discoverable, understandable, trustworthy, and supportable in search and AI answers.
• Interaction readiness: allowing an agent to navigate, read, compare, submit, book, or call tools safely and reliably.

LLMs.txt may contribute to the second goal for systems that adopt it. Google explicitly says it contributes nothing special to the first goal inside Google Search.

The Nugentive Recommendation

Do not delete LLMs.txt because John Mueller was skeptical. Do not install it because Lighthouse displayed a warning. Use the evidence.

For a business with solid content, clear services, good proof, reliable hosting, and a maintained website, adding a concise root LLMs.txt file is a reasonable low-cost experiment. Log requests. Keep it accurate. Re-test it. Just do not sell the owner a story about rankings that Google has already denied.

For a business with weak service pages, inconsistent information, poor reviews, crawler blocks, inaccessible forms, or unstable layouts, LLMs.txt belongs lower on the list. Fix what customers and widely used systems already depend on.

That is the practical value of an AI Visibility Audit: it separates emerging experiments from the blockers that are costing attention, trust, and customers now. You get a prioritized fix list based on what systems can access, understand, verify, cite, and use—not another shiny file uploaded for ceremonial purposes.

(/ai-visibility-audit)