41.3% of Websites Don't List Their XML Sitemap in Robots.txt

Every time a search engine arrives at a website, the first file it asks for is robots.txt, a short plain-text file at the root of the domain that says which parts of the site crawlers may visit. That file can also carry one more line: "Sitemap:", followed by the address of the site's XML sitemap, the machine-readable list of every page the owner wants found.

That one line matters because it works for every crawler at once. Submitting a sitemap in Google Search Console tells Google, but not Bing, not AI search crawlers and not the many smaller engines that feed other tools. The Sitemap line in robots.txt tells all of them, in the one file they all read first, and it costs nothing to add.

Four in ten websites skip it. 41.3% of websites don't list an XML sitemap in robots.txt. The surprise is who skips it: popular websites leave the line out more often than small ones, and the middle of the web, sites ranked between 10k and 1 million by traffic, leaves it out more than half the time.

We read the robots.txt files of 15,470,478 websites in the public HTTP Archive crawl of September 2026. For each website we recorded whether the file loaded and whether it contained a Sitemap line. We then grouped websites by popularity using Google's Chrome UX Report, which ranks sites by how many Chrome users visit them: the 10k most visited websites, websites ranked between 10k and 1 million, and websites outside the top million.

A website counts as listing its sitemap when its robots.txt file contains at least one line starting with "Sitemap:". A working robots.txt file is one that loads normally when a browser or crawler requests it. An XML sitemap is the file, usually at /sitemap.xml, that lists the pages a site wants search engines to find and index.

That is 6,387,827 websites that make every crawler find their sitemap some other way. The other 58.7% point to it directly. Submitting a sitemap to Google doesn't close the gap, because that submission is private to Google: every other search engine and AI crawler still has to guess, usually by trying /sitemap.xml and hoping.

A sitemap matters most for the pages a crawler would not otherwise find: new pages, pages few other pages link to, and pages deep inside a large site. Leaving it unlisted doesn't stop a site from being indexed, but it makes discovery slower and less complete for every engine that isn't Google.

The pattern runs against what most people would expect. Websites outside the top million, the small business sites, blogs and local shops that make up most of the web, do best, with 40.6% missing the line. The 10k most visited websites miss it 45.9% of the time, and the middle group, ranked between 10k and 1 million, misses it 53.5% of the time.

One likely reason is who writes the file. Many small websites run on platforms such as WordPress, Shopify, Wix or Squarespace, which generate robots.txt automatically and usually add the Sitemap line for the owner. Larger websites more often run custom builds where a developer writes robots.txt by hand, and a line that has no visible effect on the site is easy to forget. Size and budget don't guarantee the basics.

Only part of the gap comes from websites with no file at all. 13.7% of websites have no working robots.txt. The bigger share of the problem sits with websites that have the file and leave the line out: 32% of websites with a working robots.txt don't list a sitemap in it.

That makes the fix trivial for most of these websites, because the file already exists and someone already edits it. The most visited websites show it clearly: 6,620 of the 7,500 top-10k websites have a working robots.txt, yet only 4,055 of them use it to point to a sitemap. For thousands of heavily visited websites, the missing piece is a single line of text.

Open yourdomain.com/robots.txt in a browser. If no line starts with "Sitemap:", add one that gives the full address of your sitemap, such as "Sitemap: https://www.yourdomain.com/sitemap.xml". If you run several sitemaps, list each one on its own line, or list the sitemap index file that links to them.

Then check the sitemap itself: it should load, list only pages you want indexed, and leave out redirected, deleted and blocked pages. Submit the same address in Google Search Console and Bing Webmaster Tools so both engines report errors back to you. If your site runs on a custom build, add robots.txt and the sitemap to your release checklist so a redesign or migration doesn't silently drop them. This is the kind of basic setting our technical seo services check first, because it takes two minutes to fix and affects every search engine at once.

Embed this research

Cite this study