LLMS Reader Network tools

What is llms.txt?

llms.txt is a proposal by Jeremy Howard (Answer.AI) for a single markdown file at /llms.txt that tells a language model what a site contains and where to find the clean version of each page. It exists because the thing an assistant needs and the thing a browser needs are no longer the same: a modern page ships navigation, cookie banners, analytics and framework noise around a few hundred words of actual content, and a model reading that HTML spends most of its context window on markup it will throw away. The file is a curated index, written for machines but perfectly readable by people.

It is not a standard, not enforced by anyone, and not a ranking signal. No search engine has committed to it. What it does have is adoption: a large share of developer-facing documentation sites now publish one, because it is cheap to generate and it measurably improves how assistants answer questions about their product.

llms.txt vs llms-full.txt vs robots.txt

The distinction that matters in practice: llms.txt is an index and llms-full.txt is the whole corpus inlined. A site can publish either, both, or neither. This tool checks for both, and when the full file exists it loads on demand and is presented as a collapsible outline — opening a 6 MB text file in a browser tab is not a reading experience.

The format

The proposal asks for a small, fixed shape. Everything after the title is optional:

# Project name

> One-paragraph summary of what this project is.

Any free-form markdown can go here — context, conventions, warnings.

## Docs

- [Quickstart](https://example.com/quickstart.md): Get running in five minutes
- [API reference](https://example.com/api.md): Every endpoint, with examples

## Optional

- [Changelog](https://example.com/changelog.md): Release history

Three rules carry most of the weight. There is exactly one # H1 with the project name. The > blockquote directly under it is the summary a model reads first. Every ## section holds a bulleted list of markdown links, each optionally followed by a colon and a short description — and a section literally named Optional marks links that may be skipped when context is tight. This tool checks each of those and reports what it found.

How the GEO score works

GEO — generative engine optimization — is the same instinct as SEO pointed at assistants instead of search results. The score on this page is our own heuristic, not a standard: llms.txt has no scoring body, no certification and no official rubric, and anyone who tells you their number is authoritative is selling something. What it is, honestly, is a checklist with weights attached, built from the proposal itself and from what actually helps a model answer questions about a site. Use it to find the gaps, not to compare yourself with a competitor to the decimal point.

34 checks in 5 groups, weighted to add up to exactly 100 points. Every one of them runs on what a handful of requests already returned — the file itself, your homepage and your robots.txt. There is no crawl, and no request per link: a file with 629 links would take minutes.

The weights are the argument, so they are written down in one place rather than scattered through the code: content is half the score, because what your links say is the thing an assistant actually reads. The thresholds are deliberately strict, because a checklist everyone passes tells you nothing — full marks for descriptions means every link, and full marks for clean markdown copies means all of them. Scoring in the eighties here is a good file; the nineties means you have done work most sites have not.

Two rules about what a zero means, and they are not the same rule. A check we could not measure is dropped from the total, not scored zero — if a bot filter blocks our request to your homepage, the search-engine checks are marked “not scored” and the page tells you the total shrank, because punishing a site for a failed measurement would be dishonest. A check you simply did not earn scores zero and stays in the denominator. And an empty set earns nothing: a file with no links does not get full marks for “all of its titles being meaningful”.

The line between those two is whether the thing blocking the measurement has a reason of its own. Protecting your homepage with a bot filter is a legitimate security choice, so those checks are dropped and the rest of your score is scaled up — which is the same as assuming you would have scored your own average on them. Not sending a Last-Modified header is simply an omission that buys you nothing, so it scores zero and stays in the total. The honest caveat: scaling gives a small advantage to a site that would have scored below its own average on the part we could not reach. That is the standard way to handle missing data, and the alternative — scoring a blocked site zero — would be a worse lie.

One caveat worth knowing: the “clean markdown copies” check detects .md and .txt by file extension. A site that serves markdown at extensionless URLs through content negotiation is doing the right thing and will still score low there. That is a limit of the check, not a fault in the site.

How this reader works

1. You type a domain. The tool requests https://<domain>/llms.txt first.
2. If that misses, it tries the www variant, then docs. and developers. — documentation subdomains publish these far more often than the marketing root does. Every address it tried is listed in the result, so nothing is guessed silently.
3. The response is checked before it is trusted. A parked domain or a single-page app will happily answer HTTP 200 with an HTML error page; that is reported as a miss rather than rendered as if it were a real file.
4. The markdown is parsed into a title, summary, sections and link lists, and rendered as cards with a live filter. The raw source is shown separately with syntax colouring and line numbers.
5. All fetching happens server-side through our SSRF-guarded HTTP layer: private and internal addresses are rejected, redirects are re-validated at every hop, and the response size is capped.

Where these rules come from

The structural checks follow the proposal itself: an H1 is the only required element, the blockquote summary is optional, the body takes any markdown except headings, and only an H2 starts a link list. The proposal also asks for markdown copies of pages, declared with <link rel="alternate" type="text/markdown"> — which is why this tool looks for that tag rather than only guessing .md URLs.

The quality thresholds come from practice rather than the spec, and we say so where it matters. The 10–40 link range is the range community guides converge on, with effectiveness dropping past roughly fifty because the model has to scan the whole file and every entry gets diluted. The warning about generated files is the same observation from the other side: dumping a sitemap into markdown produces bulk without curation, and curation is the entire value of the format. Worth knowing: llms-full.txt appears nowhere in the proposal — it is a widely-copied community convention, which is why it sits in the smallest scoring group here rather than the structural ones.

Frequently asked questions