Does Your B2B Website Need an llms.txt? What 300,000 Domains and Our Own Bot Log Show

The question now comes up regularly in first calls, usually from a marketing lead who was just sold an llms.txt as a GEO measure, often alongside an audit report flagging the missing file in red. In Germany alone the term gets roughly 2,400 searches a month, so the confusion is real.

Short answer with evidence: for a B2B marketing website, llms.txt is not a visibility lever. And because you should not take our word for it, there is a five-minute check for your own domain below.

What an llms.txt is

The idea comes from Jeremy Howard (Answer.AI) in September 2024: a Markdown file at /llms.txt that summarises a website in a few paragraphs and links to its most important pages. The reasoning is sound. Language models have a limited context window, and HTML is full of navigation, cookie banners and script clutter. Curated Markdown saves the model work.

One thing matters for context: llms.txt is a proposal, not an adopted standard. Unlike robots.txt, no search engine and no AI vendor has publicly committed to reading it. It only works if something actually fetches it, and that part is measurable.

What the data says

300,000 domains, no measurable effect on citations

On 7 November 2025, SE Ranking published an analysis of roughly 300,000 domains (Yulia Deda and Svitlana Tomko). 10.13% of them served an llms.txt. The distribution is telling: adoption is essentially flat across traffic tiers, 9.88% for domains with 0 to 100 visits, 10.54% for 1,001 to 5,000, and 8.27% for 100,001 and above. Adoption, the authors write, "isn't concentrated among industry leaders".

The second half is more interesting. The team used Spearman correlation, XGBoost regression and SHAP analysis to test how much a present llms.txt explains how often a domain gets cited in LLM answers: "When we removed the LLMs.txt factor, the model's predictions actually improved." The model got better without the factor. Their conclusion: "LLMs.txt doesn't seem to directly impact AI citation frequency."

97% of these files are never fetched

The second study answers the blunter question: does anything fetch them at all? On 15 June 2026, Ahrefs analysed log data from 137,210 domains in its own Web and Bot Analytics (Louise Linehan, May 2026 data). The headline finding: "97% of llms.txt files receive zero traffic in May 2026. Nothing fetched them at all."

The remaining 3% is where it gets uncomfortable. The largest share of those fetches, 21.7%, comes from SEO audit tools: the very tools that then report a missing llms.txt as a defect. Behind them: unknown bots (14.9%), general crawlers (13.1%), tech-profiling tools (11.6%) and AI agents plus infrastructure (10.5%). The category everyone actually cares about is the smallest one. AI retrieval bots such as PerplexityBot and OAI-SearchBot account for 1.1% combined, AI assistants for 2.5%. Claude-Code, the coding agent, fetched more llms.txt files than all retrieval bots and assistants put together. And for domains without the file: "Zero requests came from AI bots for llms.txt files that don't exist. They never go looking."

Google says it outright

On 15 June 2026, Google added a clarification on llms.txt files to the mythbusting section of its AI optimization guide; the entry is logged in the official Search Central documentation updates. The wording leaves little room: "You don't need to create new machine readable files, AI text files, markup, or Markdown to appear in Google Search (including its generative AI capabilities), as Google Search itself doesn't use them." And: "Doing so will neither harm nor help your site's visibility or rankings in Google Search, as Google Search ignores them."

The parenthetical is the load-bearing part. AI Overviews and AI Mode are precisely those generative capabilities of Google Search. Publishing an llms.txt to show up in AI Overviews means optimising for a surface that explicitly ignores the file.

Our own bot log

That still leaves the objection that studies measure other people's domains. So we looked at our own logs, starting with fento.ai. The domain has served an llms.txt the whole time, and our server-side tracking explicitly records requests to /llms.txt. How it is built is covered in our post on the AI bots reading your website. Window: 17 June to 8 September 2026, so 83 days.

In that period, 23 different AI and search crawlers made 14,556 requests. 1,969 of them went to /robots.txt and 893 to /sitemap.xml. /llms.txt got exactly one hit, and that was our own smoke test on 17 June, run to confirm that requests to the file are captured at all. From real crawlers: zero.

The rows that matter are ClaudeBot and OAI-SearchBot. ClaudeBot fetched robots.txt 524 times and sitemap.xml 524 times, and the llms.txt not once, even though it sits in the same directory and is reachable over the same path. OAI-SearchBot read robots.txt 601 times and the llms.txt never. The other 13 bots in the log, from Claude-SearchBot through PerplexityBot to DuckAssistBot, are all at zero.

One domain is not proof, so here is the cross-check on a bigger site. A B2B client website from our tracking, an industry site with roughly four times the crawler volume, runs the same setup over the same window and has likewise served a valid llms.txt throughout (HTTP 200, text/plain, about 18 KB). There, 25 different crawlers made 62,218 requests, 4,353 of them to robots.txt and 887 to sitemap.xml. OAI-SearchBot alone fetched robots.txt 1,254 times; ClaudeBot 715 times, plus sitemap.xml 536 times. The bots even guessed at sitemap filenames that do not exist on that domain: /sitemap_index.xml, /sitemaps.xml, /news-sitemap.xml. Fetches of the llms.txt, again excluding our two smoke-test hits on 17 June: zero.

That is the point. Across both sites and 76,774 requests, these crawlers know exactly where machine readable files live. They fetch them diligently and will even guess at filenames. They just never go looking for this one.

What llms.txt is actually good for

There is one real use case. In the Ahrefs data, the most active fetcher is a coding agent. That matches who maintains the file seriously: Anthropic, Cloudflare, Stripe and Cursor all publish llms.txt files for their developer documentation. An agent asked to integrate an API gets there faster with a curated Markdown link list than with a searchable docs UI.

There is also a nuance that explains why the confusion persists. Since 5 May 2026, Chrome Lighthouse has an agentic browsing audit category that checks whether an llms.txt exists at the domain root and flags a present file when it is broken: no H1, too short, no links. So the same company that says Search ignores the file now lints it in its own browser tooling. That is a question of scope, not a contradiction: the audit targets agentic browsers and coding agents, not Search. It does reliably put someone in a marketing meeting holding a red Lighthouse item.

The adoption numbers support staying calm. Rankability tracks monthly which of the Tranco top 1,000 sites serve a valid llms.txt. In June 2026 it was 8.7%, or 87 sites, up from 0.3% in June 2025. Adoption is growing, but nearly two years in, over 90% of the largest websites in the world still do not serve one.

The honest read: if you run developer documentation, a curated llms.txt is cheap and useful. If you run a B2B marketing website, it is a nice-to-have with no visibility effect.

Check it yourself in five minutes

You do not have to take our word for it. Run this against your own domain:

1. Does the file even exist? One request tells you whether it returns status 200 and is served as text.
2. Is anything fetching it? Search your server log or bot tracking for the last 30 days by filename and group by user agent.
3. Compare against robots.txt and sitemap.xml. This is the actual test. Those two numbers show how eagerly crawlers reach for control files on your site in the first place.
4. Decide. If robots.txt shows a three-digit number and llms.txt shows a zero, you have your answer.

# 1. Does the domain serve an llms.txt at all?
curl -sI https://your-domain.com/llms.txt | head -n 5

# 2. Who fetched it in the last 30 days?
grep "llms.txt" access.log | awk -F'"' '{print $6}' | sort | uniq -c | sort -rn

# 3. For comparison: robots.txt and sitemap.xml
grep -c "robots.txt" access.log
grep -c "sitemap.xml" access.log

A note for anyone on Vercel, Netlify or Cloudflare Pages: a classic access log is often out of reach there, and Google Analytics will not help, because bots do not execute JavaScript and therefore never show up in GA. In that case you need the server-side tracking from the post linked above, roughly 50 lines of middleware that check every request against known AI user agents. After that you have the same table as above for your own domain.

What actually moves citations

If the file is not the lever, what is? Three things, in this order:

• Content that answers the questions people actually ask. AI search breaks a user question into several sub-questions and searches for each one separately. Optimise for the head keyword only and you miss most of those queries. How that mechanism works is covered in our post on fan-out queries in AI search.
• Clean, crawlable HTML. No text loaded in by JavaScript, a clear heading hierarchy, reachable internal links. Bots take the raw HTML, and what is not in there does not exist for them.
• Measuring instead of guessing. Search Console and Bing Webmaster Tools now show real AI citations from real queries. How to read those reports is covered in our post on AI search visibility in Google and Bing.

Publishing an llms.txt is not wrong. It does no harm, it costs half an hour, and if it turns a Lighthouse item green, your team gets some peace. It only becomes wrong when it takes the slot in the quarterly plan that content, structure and measurement should occupy. The file is a bet on a future nobody has cashed in yet. Your visibility in AI answers is being decided somewhere else today.

We measure which bots really read a website for B2B teams in tech and industry, and rebuild the site so it shows up in AI answers and brings in qualified inbound leads. How we work is on the services page.

Sources: SE Ranking: LLMs.txt study · Ahrefs: llms.txt study · Google: AI optimization guide, mythbusting · Google Search Central: documentation updates · Search Engine Land: Lighthouse agentic browsing audit · Chrome: Lighthouse agentic browsing scoring · Rankability: llms.txt adoption tracker · llmstxt.org: the specification