--- title: "XML Sitemap Statistics: 41.3% Skip It in Robots.txt" url: https://www.suffdigital.com/resources/data-studies/xml-sitemap-robots-txt-rate description: "XML sitemap statistics: 41.3% of websites don't list their sitemap in robots.txt, and 45.9% of the 10k most visited sites leave it out. See the data." lang: en --- Share Share on X: https://twitter.com/intent/tweet?url=https%3A%2F%2Fwww.suffdigital.com%2Fresources%2Fdata-studies%2Fxml-sitemap-robots-txt-rate&text=41.3%25%20of%20Websites%20Don%27t%20List%20Their%20XML%20Sitemap%20in%20Robots.txt Share on Facebook: https://www.facebook.com/sharer/sharer.php?u=https%3A%2F%2Fwww.suffdigital.com%2Fresources%2Fdata-studies%2Fxml-sitemap-robots-txt-rate Share on LinkedIn: https://www.linkedin.com/sharing/share-offsite/?url=https%3A%2F%2Fwww.suffdigital.com%2Fresources%2Fdata-studies%2Fxml-sitemap-robots-txt-rate Share on Reddit: https://www.reddit.com/submit?url=https%3A%2F%2Fwww.suffdigital.com%2Fresources%2Fdata-studies%2Fxml-sitemap-robots-txt-rate&title=41.3%25%20of%20Websites%20Don%27t%20List%20Their%20XML%20Sitemap%20in%20Robots.txt **41.3%** of websites don't point search engines to their XML sitemap in robots.txt, a one-line fix, and the most visited sites skip it more often than small ones: 45.9% of the 10k most visited websites leave it out, against 40.6% of sites outside the top million. Analysis of 15,470,478 websites · the public HTTP Archive crawl, September 2026 ## Key Findings 1. 1. **41.3%** of websites don't list an XML sitemap, the file that lists a site's pages for search engines, in their robots.txt file. Copy stat 2. 2. **32%** of websites that have a working robots.txt file still leave out the one Sitemap line that points search engines to their page list. Copy stat 3. 3. **13.7%** of websites have no working robots.txt file at all, the plain-text file search engines read before crawling a site. Copy stat 4. 4. **45.9%** of the 10k most visited websites don't list an XML sitemap in robots.txt, more than the 40.6% of websites outside the top million. Copy stat 5. 5. **53.5%** of websites ranked between 10k and 1 million by traffic don't list an XML sitemap in robots.txt, the worst rate of any popularity group. Copy stat ## Summary Every time a search engine arrives at a website, the first file it asks for is robots.txt, a short plain-text file at the root of the domain that says which parts of the site crawlers may visit. That file can also carry one more line: "Sitemap:", followed by the address of the site's XML sitemap, the machine-readable list of every page the owner wants found. That one line matters because it works for every crawler at once. Submitting a sitemap in Google Search Console tells Google, but not Bing, not AI search crawlers and not the many smaller engines that feed other tools. The Sitemap line in robots.txt tells all of them, in the one file they all read first, and it costs nothing to add. Four in ten websites skip it. **41.3%** of websites don't list an XML sitemap in robots.txt. The surprise is who skips it: popular websites leave the line out more often than small ones, and the middle of the web, sites ranked between 10k and 1 million by traffic, leaves it out more than half the time. ## What we measured We read the robots.txt files of 15,470,478 websites in the public HTTP Archive crawl of September 2026. For each website we recorded whether the file loaded and whether it contained a Sitemap line. We then grouped websites by popularity using Google's Chrome UX Report, which ranks sites by how many Chrome users visit them: the 10k most visited websites, websites ranked between 10k and 1 million, and websites outside the top million. A website counts as listing its sitemap when its robots.txt file contains at least one line starting with "Sitemap:". A working robots.txt file is one that loads normally when a browser or crawler requests it. An XML sitemap is the file, usually at /sitemap.xml, that lists the pages a site wants search engines to find and index. ## Four in ten websites leave the sitemap line out 41.3% of websites don't list an XML sitemap in robots.txt. That is 6,387,827 websites that make every crawler find their sitemap some other way. The other 58.7% point to it directly. Submitting a sitemap to Google doesn't close the gap, because that submission is private to Google: every other search engine and AI crawler still has to guess, usually by trying /sitemap.xml and hoping. A sitemap matters most for the pages a crawler would not otherwise find: new pages, pages few other pages link to, and pages deep inside a large site. Leaving it unlisted doesn't stop a site from being indexed, but it makes discovery slower and less complete for every engine that isn't Google. ## Popular websites skip the sitemap line more than small ones 53.5% of websites ranked between 10k and 1 million by traffic don't list an XML sitemap in robots.txt. The pattern runs against what most people would expect. Websites outside the top million, the small business sites, blogs and local shops that make up most of the web, do best, with 40.6% missing the line. The 10k most visited websites miss it 45.9% of the time, and the middle group, ranked between 10k and 1 million, misses it 53.5% of the time. No sitemap in robots.txt by website group 10k most visited websites 45.9% Ranked 10k to 1 million 53.5% Outside the top million 40.6% All websites 41.3% Source: Suff Digital analysis of 15,470,478 websites · the public HTTP Archive crawl, September 2026 One likely reason is who writes the file. Many small websites run on platforms such as WordPress, Shopify, Wix or Squarespace, which generate robots.txt automatically and usually add the Sitemap line for the owner. Larger websites more often run custom builds where a developer writes robots.txt by hand, and a line that has no visible effect on the site is easy to forget. Size and budget don't guarantee the basics. ## Most misses come from sites that already have a robots.txt file Only part of the gap comes from websites with no file at all. 13.7% of websites have no working robots.txt. The bigger share of the problem sits with websites that have the file and leave the line out: 32% of websites with a working robots.txt don't list a sitemap in it. Share of websites No sitemap listed in robots.txt 41.3% Robots.txt present but no Sitemap line 32% No working robots.txt 13.7% That makes the fix trivial for most of these websites, because the file already exists and someone already edits it. The most visited websites show it clearly: 6,620 of the 7,500 top-10k websites have a working robots.txt, yet only 4,055 of them use it to point to a sitemap. For thousands of heavily visited websites, the missing piece is a single line of text. ## What this means for website owners Open yourdomain.com/robots.txt in a browser. If no line starts with "Sitemap:", add one that gives the full address of your sitemap, such as "Sitemap: https://www.yourdomain.com/sitemap.xml". If you run several sitemaps, list each one on its own line, or list the sitemap index file that links to them. Then check the sitemap itself: it should load, list only pages you want indexed, and leave out redirected, deleted and blocked pages. Submit the same address in Google Search Console and Bing Webmaster Tools so both engines report errors back to you. If your site runs on a custom build, add robots.txt and the sitemap to your release checklist so a redesign or migration doesn't silently drop them. This is the kind of basic setting our technical seo services (https://www.suffdigital.com/seo/technical-seo-services) check first, because it takes two minutes to fix and affects every search engine at once. ## Embed this research Paste this on your site to embed the charts. It links back to the source automatically.

Source: 41.3% of Websites Don't List Their XML Sitemap in Robots.txt - Suff Digital

## Cite this study APA Suff Digital. (2026). 41.3% of Websites Don't List Their XML Sitemap in Robots.txt. https://www.suffdigital.com/resources/data-studies/xml-sitemap-robots-txt-rate Plain link 41.3% of Websites Don't List Their XML Sitemap in Robots.txt - Suff Digital - https://www.suffdigital.com/resources/data-studies/xml-sitemap-robots-txt-rate ## Frequently asked questions ### What percentage of websites list an XML sitemap in robots.txt? ### Should you list your sitemap in robots.txt? ### Do popular websites list their sitemaps in robots.txt? ### How many websites have no robots.txt file? ## Related studies - SEO 18.3% of Canonical Tags on Top Website Homepages Point Google to a Different URL: https://www.suffdigital.com/resources/data-studies/canonical-tags-point-elsewhere - SEO Two Thirds of Website Homepages Fail at Least One Basic SEO Check: https://www.suffdigital.com/resources/data-studies/websites-fail-basic-seo-checks - SEO 49.6% of Homepages Have No H1 Heading or More Than One: https://www.suffdigital.com/resources/data-studies/homepages-missing-h1-headings - SEO Followed Backlinks to Small Business Websites Vanish Five Times as Often as Nofollow Links: https://www.suffdigital.com/resources/data-studies/backlink-link-rot-rate ## Structured data ```json [ { "@context": "https://schema.org", "@graph": [ { "@type": "Organization", "@id": "https://www.suffdigital.com/#organization", "name": "Suff Digital", "alternateName": "Suff Digital Marketing", "url": "https://www.suffdigital.com/", "logo": { "@type": "ImageObject", "@id": "https://www.suffdigital.com/#logo", "url": "https://www.suffdigital.com/logo.png", "contentUrl": "https://www.suffdigital.com/logo.png", "caption": "Suff Digital" }, "image": { "@id": "https://www.suffdigital.com/#logo" }, "description": "Suff Digital is a full-stack digital marketing agency that turns websites into predictable revenue engines, SEO, paid media, content, CRO, and web design built around revenue, not vanity metrics.", "slogan": "Digital marketing built for what's next.", "email": "hello@suffdigital.com", "telephone": "+1-629-236-0099", "foundingDate": "2022", "founder": { "@type": "Person", "name": "Matt Suffoletto", "jobTitle": "Founder & CEO" }, "contactPoint": [ { "@type": "ContactPoint", "telephone": "+1-629-236-0099", "email": "hello@suffdigital.com", "contactType": "sales", "areaServed": "US", "availableLanguage": [ "en" ] } ], "address": { "@type": "PostalAddress", "addressLocality": "Nashville", "addressRegion": "TN", "postalCode": "37219", "addressCountry": "US" }, "areaServed": { "@type": "Country", "name": "United States" }, "sameAs": [ "https://www.linkedin.com/company/suff-digital/", "https://www.instagram.com/suffdigital/", "https://www.youtube.com/@suffdigitalmarketing", "https://www.tiktok.com/@suffdigital" ], "knowsAbout": [ "Search Engine Optimization", "Pay-Per-Click Advertising", "Paid Social Advertising", "Conversion Rate Optimization", "Web Design", "Digital PR", "Link Building", "Website Migration", "Website Maintenance and Support", "Technical SEO", "Local SEO", "AI Search Optimization" ], "hasOfferCatalog": { "@type": "OfferCatalog", "name": "Suff Digital Services", "itemListElement": [ { "@type": "Offer", "itemOffered": { "@type": "Service", "name": "Search Engine Optimization (SEO)", "url": "https://www.suffdigital.com/" } }, { "@type": "Offer", "itemOffered": { "@type": "Service", "name": "Pay-Per-Click (PPC) Advertising", "url": "https://www.suffdigital.com/services/ppc" } }, { "@type": "Offer", "itemOffered": { "@type": "Service", "name": "Paid Social Advertising", "url": "https://www.suffdigital.com/services/paid-social" } }, { "@type": "Offer", "itemOffered": { "@type": "Service", "name": "Conversion Rate Optimization (CRO)", "url": "https://www.suffdigital.com/services/cro" } }, { "@type": "Offer", "itemOffered": { "@type": "Service", "name": "Web Design", "url": "https://www.suffdigital.com/services/web-design" } }, { "@type": "Offer", "itemOffered": { "@type": "Service", "name": "Digital PR & Link Building", "url": "https://www.suffdigital.com/services/digital-pr" } }, { "@type": "Offer", "itemOffered": { "@type": "Service", "name": "Website Migration", "url": "https://www.suffdigital.com/services/website-migration" } }, { "@type": "Offer", "itemOffered": { "@type": "Service", "name": "Website Maintenance & Support", "url": "https://www.suffdigital.com/services/website-maintenance" } } ] } }, { "@type": "WebSite", "@id": "https://www.suffdigital.com/#website", "url": "https://www.suffdigital.com/", "name": "Suff Digital", "inLanguage": "en-US", "publisher": { "@id": "https://www.suffdigital.com/#organization" } } ] }, { "@context": "https://schema.org", "@graph": [ { "@type": "Article", "@id": "https://www.suffdigital.com/resources/data-studies/xml-sitemap-robots-txt-rate#article", "headline": "41.3% of Websites Don't List Their XML Sitemap in Robots.txt", "description": "41.3% of websites don't point search engines to their XML sitemap in robots.txt, a one-line fix, and the most visited sites skip it more often than small ones: 45.9% of the 10k most visited websites leave it out, against 40.6% of sites outside the top million.", "image": [ "https://www.suffdigital.com/images/suff-digital-share-16x9.jpg" ], "url": "https://www.suffdigital.com/resources/data-studies/xml-sitemap-robots-txt-rate", "datePublished": "2026-09-24", "dateModified": "2026-09-24", "author": { "@type": "Person", "name": "Matt Suffoletto", "jobTitle": "Founder & CEO", "url": "https://www.suffdigital.com/who-we-are" }, "publisher": { "@id": "https://www.suffdigital.com/#organization" }, "mainEntityOfPage": { "@id": "https://www.suffdigital.com/resources/data-studies/xml-sitemap-robots-txt-rate", "@type": "WebPage" }, "isPartOf": { "@id": "https://www.suffdigital.com/#website" } }, { "@type": "BreadcrumbList", "@id": "https://www.suffdigital.com/resources/data-studies/xml-sitemap-robots-txt-rate#breadcrumb", "itemListElement": [ { "@type": "ListItem", "position": 1, "name": "Home", "item": "https://www.suffdigital.com/" }, { "@type": "ListItem", "position": 2, "name": "Resources", "item": "https://www.suffdigital.com/resources" }, { "@type": "ListItem", "position": 3, "name": "Data Studies", "item": "https://www.suffdigital.com/resources/data-studies" }, { "@type": "ListItem", "position": 4, "name": "41.3% of Websites Don't List Their XML Sitemap in Robots.txt" } ] }, { "@type": "Dataset", "@id": "https://www.suffdigital.com/resources/data-studies/xml-sitemap-robots-txt-rate#dataset", "name": "41.3% of Websites Don't List Their XML Sitemap in Robots.txt", "description": "41.3% of websites don't point search engines to their XML sitemap in robots.txt, a one-line fix, and the most visited sites skip it more often than small ones: 45.9% of the 10k most visited websites leave it out, against 40.6% of sites outside the top million.", "url": "https://www.suffdigital.com/resources/data-studies/xml-sitemap-robots-txt-rate", "creator": { "@id": "https://www.suffdigital.com/#organization" }, "datePublished": "2026-09-24", "dateModified": "2026-09-24", "license": "https://creativecommons.org/licenses/by/4.0/", "isAccessibleForFree": true, "measurementTechnique": "The public HTTP Archive crawl (robots.txt files); Google's Chrome UX Report popularity ranks" }, { "@type": "FAQPage", "@id": "https://www.suffdigital.com/resources/data-studies/xml-sitemap-robots-txt-rate#faq", "mainEntity": [ { "@type": "Question", "name": "What percentage of websites list an XML sitemap in robots.txt?", "acceptedAnswer": { "@type": "Answer", "text": "58.7% of websites list an XML sitemap in their robots.txt file and 41.3% don't, based on 15,470,478 websites in the September 2026 HTTP Archive crawl." } }, { "@type": "Question", "name": "Should you list your sitemap in robots.txt?", "acceptedAnswer": { "@type": "Answer", "text": "Yes. One \"Sitemap:\" line tells every search engine and AI crawler where your page list is, yet 32% of websites with a working robots.txt file leave it out." } }, { "@type": "Question", "name": "Do popular websites list their sitemaps in robots.txt?", "acceptedAnswer": { "@type": "Answer", "text": "Less often than small ones. 45.9% of the 10k most visited websites don't list an XML sitemap in robots.txt, against 40.6% of websites outside the top million." } }, { "@type": "Question", "name": "How many websites have no robots.txt file?", "acceptedAnswer": { "@type": "Answer", "text": "13.7% of websites have no working robots.txt file, the plain-text file search engines read before crawling a site." } } ] } ] } ] ```